# Rittman Analytics > Boutique data and AI analytics consultancy building modern data platforms, warehouse migrations and agentic analytics for mid-market, high-growth businesses in the UK, EU and USA. Rittman Analytics designs and builds data platforms (dbt, Dagster, BigQuery, Snowflake, Looker, Cube, Fivetran), migrates legacy warehouses, enables in-house data teams and delivers AI and predictive analytics. Delivery is AI-augmented using our Wire Framework blueprint. Every link below is a Markdown copy of the page; drop the ".md" suffix for the HTML page. --- # Data Platform & AI Analytics Consultancy | Rittman Analytics Canonical: https://rittmananalytics.com/ URL: https://rittmananalytics.com/ Description: Boutique data consultancy building modern data platforms and AI-ready analytics for ambitious, fast-growing businesses. --- [Gartner® predicts](https://www.gartner.com/en/newsroom/press-releases/2025-06-17-gartner-announces-top-data-and-analytics-predictions) \>50% of business decisions will be driven by AI agents by 2027. # Your data's job just changed. But you can't plug an AI agent into messy, fragmented data nobody trusts. We design, build, and modernise the governed data stacks that power reliable reporting today and agentic analytics tomorrow. [See client results ›](https://rittmananalytics.com/case-studies) 45 minutes · no obligation · you'll speak with a founder, not sales Trusted by ambitious data teams at: [![ROUVY](https://rittmananalytics.com/lovable-uploads/17e6db82-3823-42cc-a500-f44c28b5898a.png)](https://rittmananalytics.com/case-studies/rouvy)[![Rixo](https://rittmananalytics.com/lovable-uploads/d31fb621-3b64-42a8-b80a-dcd5b349fa1c.png)](https://rittmananalytics.com/case-studies/rixo)[![Pleo](https://rittmananalytics.com/lovable-uploads/d11c5354-2d8e-4c41-9369-0c924bca69e6.png)](https://rittmananalytics.com/case-studies/pleo)[![Conversion Interactive](https://rittmananalytics.com/lovable-uploads/6fe9e326-335e-4023-9fa6-9869257b511d.png)](https://rittmananalytics.com/case-studies/conversion-interactive)[![Osprey](https://rittmananalytics.com/lovable-uploads/dde3db09-20ba-4049-ae39-0efcea408b17.png)](https://rittmananalytics.com/case-studies/osprey) ## What's your Data & AI Challenge? Every organisation's data journey is unique. Select the scenario that best describes your current needs to see how we can help. ![Build an AI-Ready Platform](https://rittmananalytics.com/lovable-uploads/8d5c61f2-915c-4cb2-984a-5cecea7314b4.png) ### Build an AI-Ready Platform You're growing fast and need a scalable data foundation that's AI ready, self-service and provides a single version-of-the-truth. [Read More \>](https://rittmananalytics.com/services/data-platform-development) ![Deliver Data and AI Insights Faster](https://rittmananalytics.com/lovable-uploads/agentic_development.png) ### Deliver Data and AI Insights Faster Our Wire Framework embeds AI into every stage of the analytics lifecycle, while keeping your expert team in complete control. [Read More \>](https://rittmananalytics.com/services/wire-framework) ![Deploy Trustable Analytics Agents](https://rittmananalytics.com/lovable-uploads/analytic_chatbots.png) ### Deploy Trustable Analytics Agents Build conversational analytics that understand your data, answer complex questions, and delivers trusted answers. [Read More \>](https://rittmananalytics.com/solutions/ai-predictive-analytics) ![Solve a Targeted Bottleneck](https://rittmananalytics.com/lovable-uploads/7ece34d7-5347-4867-baa8-18370d709c88.png) ### Solve a Targeted Bottleneck You're looking for help with Looker, dbt, Cube, BigQuery, event tracking or other modern data stack tools [Read More \>](https://rittmananalytics.com/services/technical-consulting) Why now ## Your data platform now has two customers. People need dashboards, self-service analytics and metrics they trust. AI agents need exactly the same thing, one governed, well-modelled source of the right answer. Build it once, and it serves both. Your data Product events CRM & marketing Finance & ops SaaS apps The governed platform we build Data foundation BigQuery or Snowflake, modelled with dbt, tested, documented, one place for everything Semantic layer One definition of every metric, the single right answer both customers draw from Governance Access controls, audit trails and evals, so every answer is defensible Two customers, same answer People Trusted dashboardsSelf-service analyticsNumbers that match the board pack Agents Conversational analyticsAgentic workflows via MCPAuditable, defensible answers Build the foundation once, every answer, to a person or an agent, comes from the same governed source. [Build your Data Platform](https://rittmananalytics.com/services/data-platform-development)[Migrate your Data Platform](https://rittmananalytics.com/services/data-warehouse-migration) ![BARK logo](https://rittmananalytics.com/__l5e/assets-v1/d4bb7e56-e8fb-4d5a-b755-adb0e821f514/barkbox-logo.png) > “Working with Rittman Analytics allowed us to deliver the intuitive, reliable data experience our teams have needed. Their expertise helped us accelerate adoption across the business with minimal friction, one of the smoothest consulting engagements of my career.” ![Michael Bishop, VP Data at BARK](https://rittmananalytics.com/lovable-uploads/michael-bishop-barkbox.jpeg) Michael Bishop VP, Data, BARK ![BarkBox subscription box with dog toys and treats](https://rittmananalytics.com/lovable-uploads/barkbox-hero.png) Why us ## Senior expertise, multiplied by AI and Wire. Wire is our delivery methodology: AI generates the boilerplate, models, tests, documentation, dashboard prototypes, while our consultants keep the architecture, governance and judgment. You get senior-team rigour at a pace traditional consulting can't match. - **Faster prototypes** , data models and dashboard mockups generated from your requirements, so stakeholder conversations are concrete from day one. - **Governance at speed** , every AI-generated asset passes human-led review gates, 100% explainable, semantically consistent. - **Budget on strategy, not boilerplate** , your spend goes to architecture and enablement, not repetitive low-value tasks. [Explore the Wire Framework](https://rittmananalytics.com/services/wire-framework) ![Analyst and AI assistant working from the governed semantic model](https://rittmananalytics.com/__l5e/assets-v1/206e9901-f443-47dc-bd9d-483f33106401/wire-diagram.svg) Selected Work ## Outcomes, not deliverables. [ ![Pleo](https://rittmananalytics.com/lovable-uploads/pleo_logo.png)FinTech 60% faster release cycles One monorepo replaced two disconnected toolchains. Domain analysts now ship their own Looker content with the same QA rigour as the core team, no more waiting on a central queue. Read the case study ›](https://rittmananalytics.com/case-studies/pleo)[ ![BARK](https://rittmananalytics.com/lovable-uploads/barkbox_logo.png)eCommerce 3x reporting velocity Campaign reporting that took hours of spreadsheet stitching now runs from one semantic layer. Growth Marketing gets same-day answers instead of Friday-afternoon exports. Read the case study ›](https://rittmananalytics.com/case-studies/barkbox)[ ![ROUVY](https://rittmananalytics.com/lovable-uploads/rouvy_logo.png)Sports Tech 6mo legacy stack to full rebuild Six systems held together by custom scripts and external consultants, replaced with a governed, self-maintained stack in half a year. No more waiting on outside help to change a metric. Read the case study ›](https://rittmananalytics.com/case-studies/rouvy) ## Partnering for success We specialise in Google Cloud solutions and partner with Cube, Anthropic, Dagster, dbt Labs, Fivetran, Embeddable and other selected modern data stack vendors. [About our Google Cloud partnership ›](https://rittmananalytics.com/partners/googlecloud) ![Rittman Analytics partner logos including Google Cloud, dbt Labs, Cube, Fivetran, Anthropic, Snowflake and Dagster](https://rittmananalytics.com/__l5e/assets-v1/98fcbdba-2de9-4e91-8c3c-a646fca1e44a/partner-badge-2.png) --- # About Rittman Analytics | Data & AI Consultancy Canonical: https://rittmananalytics.com/about URL: https://rittmananalytics.com/about Description: Meet the team behind Rittman Analytics and the methodology we use to build modern data platforms and AI-ready analytics. --- We Build Data Cultures. We are a boutique data analytics consultancy that helps ambitious, data-rich businesses scale and modernize their analytics capabilities. We believe that lasting data maturity is achieved by combining deep technical expertise with a commitment to enabling your team. Our approach goes beyond traditional implementation. We work as an embedded partner to build your team's skills, establish proven processes, and deliver a data foundation that drives real business value. ## Problems We Solve for Our Clients Organisations come to us when they face challenges such as: - Fragmented or unreliable data sources - Manual, time-consuming reporting processes - No single version of the truth - A bottleneck around reporting requests - Challenges in embedding a data culture Your browser does not support the video tag. ## Why Choose Rittman Analytics? In a crowded market, our approach is what sets us apart. We deliver strategic value where other consultancies only deliver code. ### Strategic Thinking, Delivered Practically We combine architectural rigour with the pragmatism to ship value fast, blending data strategy with hands-on delivery. ### Team Enablement, Not Dependency We prioritise building your internal data capability through knowledge transfer, paired development and clear documentation, so you're not left reliant on external support. ### Opinionated, Modern Stack Expertise We specialise in best-in-class tools like dbt, Looker, BigQuery and the wider modern data stack, helping clients move fast and avoid common pitfalls. ### Embedded with Your Team We work in two-week sprints alongside your product and analytics teams, with shared goals, Slack channels and dashboards, no just ticketed tasks and updates. ### Named, Senior-Led Delivery Your project is delivered by a consistent, senior team who know your business, not handed down to junior staff or rotated through a bench. ### Trusted by Technical Decision-Makers Our clients include Heads of Data, CTOs and analytics leaders who value our ability to scale their platform, roadmap their data function and challenge their thinking constructively. ## Our Team of Experts Our team is composed of seasoned data strategists, architects, and analytics engineers with decades of combined experience. ![Mark Rittman](https://rittmananalytics.com/lovable-uploads/eda4cddd-fb82-4ac7-b5cb-14f6b3fc8682.png) ### Mark Rittman Chief Executive Officer ![Lewis Baker](https://rittmananalytics.com/lovable-uploads/a9a2c64a-b17a-4ed1-9a96-45f72fbe5c41.png) ### Lewis Baker Chief Operations Officer ![Lydia Blackley](https://rittmananalytics.com/lovable-uploads/d0fd0e44-7c66-4c7a-9b2c-f7cc8c82f610.png) ### Lydia Blackley Senior Analytics Consultant ![Jordan Ilyat](https://rittmananalytics.com/lovable-uploads/e2d77159-419e-48ff-b5ba-c1cdcf5b729e.png) ### Jordan Ilyat Senior Analytics Consultant ![Olivier Dupuis](https://rittmananalytics.com/lovable-uploads/d2ea6b4a-80bc-4204-bcd5-3e920eff2d04.png) ### Olivier Dupuis Principal Consultant ![George Sanderson](https://rittmananalytics.com/assets/george-sanderson-EetrYIhN.png) ### George Sanderson Senior Analytics Consultant ![Tim Griew](https://rittmananalytics.com/assets/tim-griew-DeBhrZf2.png) ### Tim Griew Analytics Consultant ![Saverro Suseno](https://rittmananalytics.com/assets/saverro-suseno-DCznkSTS.png) ### Saverro Suseno Associate Analytics Engineer ![Alex Caldwell](https://rittmananalytics.com/assets/alex-caldwell-Bv7LFaKa.png) ### Alex Caldwell Associate Analytics Consultant ![Sadie Mitchell](https://rittmananalytics.com/lovable-uploads/e1a25287-719c-4b4b-b4fd-0ce6ddb57f27.png) ### Sadie Mitchell Finance Manager ## Frequently Asked Questions ### About Rittman Analytics ### Services & Capabilities ### Delivery & Engagement --- # Contact Rittman Analytics | Data & AI Experts Canonical: https://rittmananalytics.com/contact URL: https://rittmananalytics.com/contact Description: Get in touch with Rittman Analytics to discuss data platform development, migration, team enablement and AI analytics. --- ### Get in touch ### Address Rittman Analytics Ltd Mocatta House, Trafalgar Terrace, Brighton, BN1 4BG ### Office +44 (0)7488 884631 ### Email info@rittmananalytics.com **Registered number:** 10456203 **VAT No:** 257 2490 90 ### Find us ### Send us a message Loading contact form... --- # Privacy Policy - Rittman Analytics Canonical: https://rittmananalytics.com/privacy-policy URL: https://rittmananalytics.com/privacy-policy Description: Read the Rittman Analytics privacy policy. Learn how we collect, use, and protect your personal information when you use our data analytics consulting services. --- --- # Rittman Analytics Blog - Articles, Tutorials, Case Studies Canonical: https://rittmananalytics.com/blog URL: https://rittmananalytics.com/blog Description: Expert articles on data strategy, modern data platforms, analytics delivery, and embedded analytics from Rittman Analytics. --- Latest thoughts on data analytics, customer insights, and industry trends [See client results ›](https://rittmananalytics.com/case-studies) 45 minutes · no obligation · you'll speak with a founder, not sales Group by: Sort by: ![New!](https://rittmananalytics.com/lovable-uploads/be1c118f-63e9-492e-bb23-cbe97fe702bf.png) ![A Preview of Fully-Agentic Project Delivery using Wire, Google Cloud and Rittman Analytics](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F10%2FScreenshot-2026-10-17-at-14.45.44.png) ### A Preview of Fully-Agentic Project Delivery using Wire, Google Cloud and Rittman Analytics At Rittman Analytics we’ve recently been experimenting with using Wire, our Claude Code plugin, to run teams of autonomous agents that work alongside our consultants when delivering analytics projects. Using… markrittman·Oct 17, 2026 ![New!](https://rittmananalytics.com/lovable-uploads/be1c118f-63e9-492e-bb23-cbe97fe702bf.png) ![Migrating Tableau, Metabase and Data Studio Reports to Looker using AI, Wire & Rittman Analytics](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F10%2F1b-bl4zdtAyHn0ArbdDr7A.png) ### [Migrating Tableau, Metabase and Data Studio Reports to Looker using AI, Wire & Rittman Analytics](https://rittmananalytics.com/blog/2026/10/11/migrating-tableau-metabase-and-data-studio-reports-to-looker-using-ai-wire-rittman-analytics) One of the most-common requests we get from clients is to help them move from legacy BI tools such as Metabase, Tableau or Power BI to a modern data analytics… markrittman·Oct 11, 2026 ![New!](https://rittmananalytics.com/lovable-uploads/be1c118f-63e9-492e-bb23-cbe97fe702bf.png) ![What’s New in Wire 4.x : Multi-Agent Development, dbt Charts, Agents Schema & More](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F10%2F1ll3tZHaKFmujJAuTWqlxRQ.png) ### [What’s New in Wire 4.x : Multi-Agent Development, dbt Charts, Agents Schema & More](https://rittmananalytics.com/blog/2026/10/11/whats-new-in-wire-4-x-multi-agent-development-dbt-charts-agents-schema-more) We just released the 4.1.0 version of Wire, the plugin for Claude Code and Gemini CLI that brings our delivery blueprints, skills and best-practices to the AI-powered agentic coding tools… markrittman·Oct 11, 2026 ![Agentic Data Platform Migration using Wire, Claude Code and Rittman Analytics](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2FData-Pipeline-Diagram-selection-1-1.png%3Ffit%3D3840%252C1640%26ssl%3D1) ### [Agentic Data Platform Migration using Wire, Claude Code and Rittman Analytics](https://rittmananalytics.com/blog/2026/07/19/agentic-data-platform-migrations-using-wire-claude-code-and-rittman-analytics) Migrating an entire analytics platform including the data warehouse, data source extractions, semantic layer, BI reports, orchestration and reverse ETL syncs from one stack to another used to be the… markrittman·Jul 19, 2026 ![Making Agentic Analytics More Accurate using Anthropic’s Agentic Data Stack and the Wire Framework](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F360ff-0mllmr2anbsdy3fh2.png) ### [Making Agentic Analytics More Accurate using Anthropic’s Agentic Data Stack and the Wire Framework](https://rittmananalytics.com/blog/2026/06/11/making-agentic-analytics-more-accurate-using-anthropics-agentic-data-stack-and-the-wire-framework) Anthropic published a blog article last week, “How Anthropic enables self-service data analytics with Claude”, describing an approach to one of the hardest problems in enterprise AI: how do you… markrittman·Jun 11, 2026 ![Google Next 2026: What’s New for Looker, BigQuery, Data Platforms and Agentic Analytics](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F3358f-1vzv4jkdgans1-jnm34yfnq.jpeg) ### [Google Next 2026: What’s New for Looker, BigQuery, Data Platforms and Agentic Analytics](https://rittmananalytics.com/blog/2026/04/26/google-next-2026-whats-new-for-looker-bigquery-data-platforms-and-agentic-analytics) Lydia, Olivier Dupuis and I were representing Rittman Analytics at Google Next 2026 in Las Vegas this last week, and it certainly was an event at Google-scale. In a masterpiece… markrittman·Apr 26, 2026 ![Introducing the Wire Framework: The “Secret Sauce” Behind Our AI-Augmented Analytics Project…](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F80274-1hdwqcjqntkn-ou-tbe3kwg.png) ### [Introducing the Wire Framework: The “Secret Sauce” Behind Our AI-Augmented Analytics Project…](https://rittmananalytics.com/blog/2026/02/25/introducing-the-wire-framework-the-secret-sauce-behind-our-ai-augmented-analytics-project) If you’ve tried asking Claude or ChatGPT to generate a dbt model or a fact table definition for you, it’s impressive how fast it can produce syntactically-correct code. But building… markrittman·Feb 25, 2026 ![So, Just How Relevant is Multi-Touch Attribution to Marketers in 2026?](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2Ff030a-1tfrmp8woifje_y8pmms6gg.png) ### [So, Just How Relevant is Multi-Touch Attribution to Marketers in 2026?](https://rittmananalytics.com/blog/2026/01/28/so-just-how-relevant-is-multi-touch-attribution-to-marketers-in-2026) If you’re running marketing for a B2C business, it’s somewhat of an understatement to say you’ve probably noticed that attribution has become significantly harder over the past few years. Between… markrittman·Jan 28, 2026 ![One Person Many Roles: Designing a Unified Person Dimension in Google BigQuery](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F01%2F0.03577433557574594.png%3Ffit%3D1236%252C1044%26ssl%3D1) ### [One Person Many Roles: Designing a Unified Person Dimension in Google BigQuery](https://rittmananalytics.com/blog/2026/01/26/one-person-many-roles-designing-a-unified-person-dimension-in-google-bigquery) When building our internal data warehouse, we quickly noticed that people and companies sit at the centre of most business processes and that the same person or company often plays… markrittman·Jan 26, 2026 ![Why We’ve Tried to Replace Data Analytics Developers Every Decade Since 1974](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F01%2F0.2706437466233763.webp%3Ffit%3D1360%252C768%26ssl%3D1) ### [Why We’ve Tried to Replace Data Analytics Developers Every Decade Since 1974](https://rittmananalytics.com/blog/2026/01/19/why-weve-tried-to-replace-data-analytics-developers-every-decade-since-1974) (This article was inspired by Stephan Schwab’s excellent piece “Why We’ve Tried to Replace Developers Every Decade Since 1969” which traces the recurring dream of eliminating software developers from COBOL… markrittman·Jan 19, 2026 ![How Rittman Analytics uses AI-Augmented Project Delivery to Provide Value to Users, Faster](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F01%2F0.11539852215881174.webp%3Ffit%3D1400%252C545%26ssl%3D1) ### [How Rittman Analytics uses AI-Augmented Project Delivery to Provide Value to Users, Faster](https://rittmananalytics.com/blog/2026/01/19/how-rittman-analytics-uses-ai-augmented-project-delivery-to-provide-value-to-users-faster) Our latest case study features Barton Peveril College in Eastleigh, UK and the AI-augmented delivery approach we used to ship their Looker and Google Cloud MVP platform in December 2025. Over the past year we’ve quietly… markrittman·Jan 19, 2026 ![You Probably Don’t Need an RFP](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F01%2F0.9555669754665426.webp%3Ffit%3D1400%252C764%26ssl%3D1) ### [You Probably Don’t Need an RFP](https://rittmananalytics.com/blog/2026/01/19/you-probably-dont-need-an-rfp) Many organisations still use large, prescriptive RFPs to select partners for data platform and analytics projects. These documents often define architecture, scope, timelines and deliverables in detail before any discovery… markrittman·Jan 19, 2026 ![Claude Meets Looker: Building Smarter, Connected Analytics with Google’s MCP Toolbox](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F12%2F0.8426204116330105.webp%3Ffit%3D1400%252C654%26ssl%3D1) ### [Claude Meets Looker: Building Smarter, Connected Analytics with Google’s MCP Toolbox](https://rittmananalytics.com/blog/2025/12/08/claude-meets-looker-building-smarter-connected-analytics-with-googles-mcp-toolbox) As well as Looker’s new built-in Conversational Analytics feature, you can now connect Looker to AI assistants such as Claude from Anthropic along with agentic developer tools such as Windsurf and Claude… markrittman·Dec 8, 2025 ![From Prompts to Skills: Automating Financial and KPI Analysis in Looker with Claude Skills and MCP Toolbox](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F12%2F0.17482841878719146.webp%3Ffit%3D1400%252C892%26ssl%3D1) ### [From Prompts to Skills: Automating Financial and KPI Analysis in Looker with Claude Skills and MCP Toolbox](https://rittmananalytics.com/blog/2025/12/08/from-prompts-to-skills-automating-financial-and-kpi-analysis-in-looker-with-claude-skills-and-mcp-toolbox) A couple of weeks ago we shared an article about the new Looker MCP Toolbox and how AI assistants like Claude.ai can now use it to query data from your Looker semantic… markrittman·Dec 8, 2025 ![An Homage to Oracle Warehouse Builder, 25 Years Ahead of Its Time](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F12%2F0.9545894212803949.gif%3Ffit%3D834%252C356%26ssl%3D1) ### [An Homage to Oracle Warehouse Builder, 25 Years Ahead of Its Time](https://rittmananalytics.com/blog/2025/12/08/an-homage-to-oracle-warehouse-builder-25-years-ahead-of-its-time) It’s now apparently twenty-five years since the first release of Oracle Warehouse Builder, Oracle’s all-in-one developer tool for building Oracle Database-hosted data warehouses first released back in 2001. Since then the… markrittman·Dec 8, 2025 ![“Fivetran In Talks to Buy Dbt Labs”, reports The Information](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F1d4e2-1wljnxv_pnyfah3bqzxwkhw.png) ### [“Fivetran In Talks to Buy Dbt Labs”, reports The Information](https://rittmananalytics.com/blog/2025/09/28/fivetran-in-talks-to-buy-dbt-labs-reports-the-information) A story in The Information published on Saturday 27th September 2025 reports that “Data Startup Fivetran In Talks to Buy Dbt Labs in Multibillion Dollar Deal”. The article is behind… markrittman·Sep 28, 2025 ![Google Integrates Spectacles into Looker for Continuous Integration & Regression Testing](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F09%2F0.21588027490316408.png%3Ffit%3D1400%252C708%26ssl%3D1) ### [Google Integrates Spectacles into Looker for Continuous Integration & Regression Testing](https://rittmananalytics.com/blog/2025/09/08/google-integrates-spectacles-into-looker-for-continuous-integration-regression-testing) When you’re developing LookML views and models in Looker, it’s possible that changes you’ve made could end-up breaking downstream looks and dashboards that rely on that LookML content; you might… markrittman·Sep 8, 2025 ![How Rittman Analytics Does Financial Analytics using Looker, Fivetran and Google BigQuery](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F09%2F0.1669793402786709.png%3Ffit%3D5116%252C3680%26ssl%3D1) ### [How Rittman Analytics Does Financial Analytics using Looker, Fivetran and Google BigQuery](https://rittmananalytics.com/blog/2025/09/07/how-rittman-analytics-does-financial-analytics-using-looker-fivetran-and-google-bigquery) As you’d expect from a data analytics consultancy that runs its business on Google Cloud, Google BigQuery and Looker, we use these services to provide the financial reports and dashboards… markrittman·Sep 7, 2025 ![Rittman Analytics is Riding from London to Brighton, 14th September 2025 for the British Heart Foundation](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F09%2F0.2040330260704829.jpeg%3Ffit%3D1344%252C768%26ssl%3D1) ### [Rittman Analytics is Riding from London to Brighton, 14th September 2025 for the British Heart Foundation](https://rittmananalytics.com/blog/2025/09/02/rittman-analytics-is-riding-from-london-to-brighton-14th-september-2025-for-the-british-heart-foundation) Six of the Rittman Analytics team are swapping SQL for saddles and Looker for Lycra as we cycle from London to Brighton on Sunday, 14th September 2025 in support of… markrittman·Sep 2, 2025 ![Opinionated, AI-Enabled & Modern: The Story Behind Rittman Analytics’ New Website](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F08%2F0.8472241509609719.png%3Ffit%3D3840%252C2785%26ssl%3D1) ### [Opinionated, AI-Enabled & Modern: The Story Behind Rittman Analytics’ New Website](https://rittmananalytics.com/blog/2025/08/29/opinionated-ai-enabled-modern-the-story-behind-rittman-analytics-new-website) Over the past few years, we’ve built and modernised data platforms for dozens of digitally native businesses. The patterns are consistent: ambitious data goals, strong technical leadership and newly formed… markrittman·Aug 29, 2025 ![Adventures in Aggregate Awareness (and Level-Specific Measures) with Looker](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F08%2F0.01426986606877012.png%3Ffit%3D2048%252C676%26ssl%3D1) ### [Adventures in Aggregate Awareness (and Level-Specific Measures) with Looker](https://rittmananalytics.com/blog/2025/08/25/adventures-in-aggregate-awareness-and-level-specific-measures-with-looker) Looker aggregate awareness is a performance optimisation feature that automatically directs users’ queries to use pre-aggregated summary tables instead of detail-level transaction tables, significantly improving the performance of reports and… markrittman·Aug 25, 2025 ![IQR-Based Website Event Anomaly Detection using Looker and Google BigQuery — Rittman Analytics](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F07%2F1753874062120-ri2bhh.webp%3Ffit%3D1536%252C1024%26ssl%3D1) ### [IQR-Based Website Event Anomaly Detection using Looker and Google BigQuery — Rittman Analytics](https://rittmananalytics.com/blog/2025/07/28/iqr-based-website-event-anomaly-detection-using-looker-and-google-bigquery-rittman-analytics) Website event tracking bugs can become costly quickly. A deployment of a new website version that triggers events multiple times can increase your hosting charges or worse, lead to overage… markrittman·Jul 28, 2025 ![Integrating Looker and LangChain with langchain-looker-agent](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2F110e5-a63820d4-6b37-4fb3-924f-52508023e9b4_1000x897.png%3Fssl%3D1) ### [Integrating Looker and LangChain with langchain-looker-agent](https://rittmananalytics.com/blog/2025/06/02/integrating-looker-and-langchain-with-langchain-looker-agent) We’ve just released a prototype of a new Langchain tool that enables LangChain applications and agentic workflows to connect to Looker, via its Open SQL Interface. In the past when… markrittman·Jun 2, 2025 ![Introducing Conversational Analytics, Data Agents and Code Interpreter in Looker and Looker Studio](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2F816ba-8f6cc46c-e352-4000-97a5-ca3022cf32c1_1000x503.png%3Fssl%3D1) ### [Introducing Conversational Analytics, Data Agents and Code Interpreter in Looker and Looker Studio](https://rittmananalytics.com/blog/2025/05/11/introducing-conversational-analytics-data-agents-and-code-interpreter-in-looker-and-lookernbspstudio) Conversational Analytics is Google’s implementation of AI-powered natural language querying for Looker and Looker Studio. At its core, it enables users to interact with their data using everyday language rather… markrittman·May 11, 2025 ![London Looker Meetup – May 8th 2025 – Registration Open & Call for Speakers!](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2Ff85d3-9e0bf430-312b-4669-84c0-3fa0a88936a4_1340x752.png%3Fssl%3D1) ### [London Looker Meetup – May 8th 2025 – Registration Open & Call for Speakers!](https://rittmananalytics.com/blog/2025/03/31/london-looker-meetup-may-8th-2025) The first London Looker Meetup will be on Thursday, May 8th at the Google London — Pancras Square office from 17:30–20:30 and registration is now open! Back in 2019 Jonathan… markrittman·Mar 31, 2025 ![How Rittman Analytics Automates Project RAG Status Reporting using Vertex AI, DocumentAI, BigQuery & Looker](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2F033e1-2b6b5931-448f-4565-97bb-a6fe006c0a00_1000x664.png%3Fssl%3D1) ### [How Rittman Analytics Automates Project RAG Status Reporting using Vertex AI, DocumentAI, BigQuery & Looker](https://rittmananalytics.com/blog/2025/01/29/genai-looker-rag-status) A common way to report on the progress of a project is through RAG (Red, Amber Green) status indicators for the project, at the overall level and by cost, schedule,… markrittman·Jan 29, 2025 ![Deploying Maxwell’s Demon: QA Automation In Droughty](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F01%2Fscreenshot2025-01-20at10.54.33.webp%3Ffit%3D1000%252C363%26ssl%3D1) ### [Deploying Maxwell’s Demon: QA Automation In Droughty](https://rittmananalytics.com/blog/2025/01/20/deploying-maxwells-demon-qa-automation-in-droughty-2) “What I value most is the trust you are given here. People make commitments to you, and they follow through. You are given the support to take ownership and do your best work.” After discovering Rittman through a Google search, Jordan joined looking for a new challenge and quickly found a team where he could grow. Nearly five years later, he has worked across a wide range of projects and enjoys the variety that comes with consulting. For Jordan, one of the biggest positives is simple: the promises made when he joined have genuinely been kept. --- # Case Studies | Rittman Analytics Canonical: https://rittmananalytics.com/case-studies URL: https://rittmananalytics.com/case-studies Description: See how leading companies modernised their data platforms and analytics with Rittman Analytics, with measurable results. --- Case Studies # Real companies. Measured outcomes. How we've helped companies across industries transform their data infrastructure, accelerate insight and drive growth. 60% faster release cycles at Pleo 3× reporting velocity at BARK 8 days SOW to live at Barton Peveril Loading case studies... --- # Data & Analytics Services | Rittman Analytics Canonical: https://rittmananalytics.com/services URL: https://rittmananalytics.com/services Description: Data platform development, warehouse migration, team enablement, embedded analytics and AI-augmented delivery services. --- We offer comprehensive data solutions tailored to your organisation's specific needs. From team transformation to technical implementation, we're your strategic partner in data excellence. ### Data Team Enablement & Transformation Transform your data team from a reactive cost centre into a proactive, strategic asset. - Strategic team assessment and diagnosis - Modern data warehouse architecture - Unified development workflows - Embedded leadership and mentorship - Cultural transformation programmes [Learn More →](https://rittmananalytics.com/services/data-team-enablement) ### Data Platform Development Build a scalable data foundation from the ground up for growing startups. - End-to-end platform architecture - Cloud infrastructure setup - Data pipeline development - BI tool implementation - Team training and handover [Learn More →](https://rittmananalytics.com/services/data-platform-development) ### Data Warehouse Migration Migrate your legacy data warehouse to a modern cloud platform using AI-powered translation. - Automated SQL and pipeline translation - BI content and dashboard migration - Schema and data sync validation - Risk-reduced cutover planning - Wire Framework acceleration [Learn More →](https://rittmananalytics.com/services/data-warehouse-migration) ### Embedded Analytics Solutions Build powerful, customer-facing analytics directly into your SaaS application. - Customer analytics dashboards - Multi-tenant architecture - API development - White-label solutions - Performance optimisation [Learn More →](https://rittmananalytics.com/services/embedded-analytics) ### Technical Consulting Expert implementation for specific challenges with dbt, Looker or event tracking. - dbt project optimisation - Looker implementation - Event tracking setup - Data quality frameworks - Performance troubleshooting [Learn More →](https://rittmananalytics.com/services/technical-consulting) --- # AI-Augmented Delivery - Move Faster Without Breaking Things | Rittman Analytics Canonical: https://rittmananalytics.com/services/ai-augmented-delivery URL: https://rittmananalytics.com/services/ai-augmented-delivery Description: Accelerate your entire data transformation lifecycle with AI as a co-pilot. Our AI-augmented methodology delivers faster insights while keeping your expert team in complete control. --- Without Breaking Things. In today's market, the pressure is relentless: deliver insights faster, but don't you dare get the numbers wrong. This forces data leaders into an impossible choice between speed and trust. We use AI as a co-pilot, not an autopilot, to accelerate your data transformation lifecycle while keeping your expert team in complete control. ## The Data Team's Dilemma: Speed vs. Trust Traditional analytics delivery is broken. Business leaders want answers yesterday, but the process is slow and fraught with risk. ### Your Challenges: - Moving too fast leads to inconsistent metrics, broken dashboards, and a gradual erosion of trust from the very stakeholders you're trying to serve. - Moving too slowly means missed opportunities, a frustrated business, and a data team that is perpetually seen as a bottleneck, not a strategic partner. - AI is not a magic bullet, used incorrectly, it just creates higher volumes of low-quality work. Used strategically, it can transform your team's entire operating model. ### Our Solution: AI-Augmented Process, Human-Led Judgment Our approach uses AI to automate repetitive tasks and accelerate feedback loops, freeing your team to focus on what matters most: complex business logic and stakeholder collaboration. #### Proof in Practice: See how we partnered with Barton Peveril College to deliver an AI-accelerated MVP analytics platform in just 15 working days, pioneering a new approach for UK education. [Read More →](https://rittmananalytics.com/case-studies/barton-peveril) ![Barton Peveril College students](https://rittmananalytics.com/lovable-uploads/barton-peveril-science-lab.png) Barton Peveril ## How AI-Augmented Delivery Works Our methodology embeds AI into every stage of the analytics lifecycle, while keeping your expert team in complete control. ![AI-generated dashboard mockup](https://rittmananalytics.com/lovable-uploads/ai-accelerated-prototyping.png) ### Accelerated Prototyping Use AI to generate initial data models and dashboard mockups directly from your requirements, making stakeholder conversations concrete and productive from day one. ![Governance and code review process](https://rittmananalytics.com/lovable-uploads/ai-governance-speed.png) ### Governance at Speed Ensure every AI-generated asset passes through clear, human-led review gates. We maintain 100% explainability, semantic clarity, and trust in your final data products. ![Accelerated AI delivery timeline](https://rittmananalytics.com/lovable-uploads/ai-faster-iteration.png) ### Faster Iteration Instantly generate boilerplate code, documentation, and tests, allowing your engineers to focus on solving unique business problems, not on repetitive, low-value tasks. ![Collaborative review and validation](https://rittmananalytics.com/lovable-uploads/ai-deeper-collaboration.png) ### Deeper Collaboration Free your team from waiting on slow builds and manual processes. More time is spent with stakeholders validating insights and less time debugging pipelines. ## From Vision to Reality: How We Helped Barton Peveril Pioneer AI-Accelerated Analytics Barton Peveril Sixth Form College needed a modern analytics platform to support strategic decision-making, but with a director demonstration scheduled for early January and just 19 working days available, they needed a partner who could deliver enterprise-quality analytics at startup speed. > "This MVP has proven what's possible when strong partnerships meet a clear vision, high-quality data, trusted metrics, and genuinely accessible analytics. I'm incredibly excited about the potential impact it can have on how data is used across education." Chris Loveday, Vice Principal, Barton Peveril College ### Our Partnership Delivered: - Complete MVP analytics platform delivered in just 15 working days - 107 automated data quality tests with 100% documentation coverage - Dimensional data warehouse in Google BigQuery with star schema architecture - Complete Looker semantic layer with 8 production dashboards - AI-powered voice and chat support bot for ongoing staff enablement ![Barton Peveril AI-Accelerated MVP Development Timeline](https://rittmananalytics.com/lovable-uploads/barton-peveril-timeline.png) [Read the Full Barton Peveril Case Study →](https://rittmananalytics.com/case-studies/barton-peveril) ## See the Future of Analytics Delivery Let us walk you through how our AI-augmented approach can be applied to your team and your current data stack. In a no-obligation conversation, we can demonstrate how to build a smarter, faster, and more collaborative analytics process. --- # Data Platform Development - Build Your Scalable Data Foundation | Rittman Analytics Canonical: https://rittmananalytics.com/services/data-platform-development URL: https://rittmananalytics.com/services/data-platform-development Description: Stop guessing. Start building. Partner with us to build a scalable data platform through collaborative implementation, knowledge transfer, and proven methodologies. From strategic discovery to agile delivery. --- Your Spreadsheets Aren't. We Can Help. You've achieved product-market fit and you're growing fast. But the cobbled-together reports and manual processes that got you here are now breaking under the strain. [See client results ›](https://rittmananalytics.com/case-studies) 45 minutes · no obligation · you'll speak with a founder, not sales We've helped build the data foundations for innovative companies like: [![Conversion Interactive](https://rittmananalytics.com/lovable-uploads/6fe9e326-335e-4023-9fa6-9869257b511d.png)](https://rittmananalytics.com/case-studies/conversion-interactive)[![Rixo](https://rittmananalytics.com/lovable-uploads/d31fb621-3b64-42a8-b80a-dcd5b349fa1c.png)](https://rittmananalytics.com/case-studies/rixo)[![Osprey](https://rittmananalytics.com/lovable-uploads/67623a6f-e148-47f5-9b96-81d4b65fb458.png)](https://rittmananalytics.com/case-studies/osprey)[![Ageras Group](https://rittmananalytics.com/lovable-uploads/fffb8292-451d-4984-934e-7a3ccd13c0ba.png)](https://rittmananalytics.com/case-studies/ageras-group)![BridgeU](https://rittmananalytics.com/lovable-uploads/fc984dbd-c774-4ed3-a6b9-959c6099b50f.png) ## You've Made Your First Data Hire. Now the Real Work Begins. You've hired your first data professional, but building a scalable data platform from scratch is a monumental task. To get it right the first time, you need a partner with a proven blueprint for success. We are that partner. ### Your Challenges: #### "DIY" Gone Wrong Your first attempts at building a data stack are improvised, often by engineers or product managers moonlighting as data people. The result? Technical debt disguised as progress. #### It's Not Time to "Learn on the Job" You know data's strategic value and want to build the right foundations, but you don't have the time or budget to let your team learn on the job. Paralysis sets in, and decisions get delayed. #### Tools ≠ Platform You've signed up for BigQuery, dbt, or Looker, but without a coherent model or strategy, they sit underused or misused. The stack exists, but the insights don't. ### Our Solution: Collaborative Data Platform Implementation We partner with you to build your scalable data platform through collaborative implementation, knowledge transfer and sharing of best practices with your team in-order to ensure long-term self-sufficiency. #### Proof in Practice: See how we enabled EV charging network Osprey scale their data analytics platform, accelerate their operational reporting and build their data team's capabilities [Read More →](https://rittmananalytics.com/case-studies/osprey) ![Osprey charging station at night](https://rittmananalytics.com/lovable-uploads/b9eb5401-ea3b-479b-bbac-e407fb8abc8c.png) Osprey ![Dagster](https://rittmananalytics.com/lovable-uploads/968ca976-3824-469d-9320-3728a0488312.png)![Snowflake](https://rittmananalytics.com/lovable-uploads/0d930e52-78f6-457e-ae69-043f540f62f4.png)![Snowcat Cloud](https://rittmananalytics.com/lovable-uploads/b414e4f8-ce94-4f89-b4d5-bcb73fe3f545.png)![Embeddable](https://rittmananalytics.com/lovable-uploads/1a278600-bf96-465b-b9d0-c126bae2b885.png) ## We Don't Just Implement Tools. We Build Your Data Foundation. For a growing business, your first data platform is one of the most critical infrastructure investments you'll make. A weak foundation built on ad-hoc decisions will accumulate technical debt and slow you down for years. A strong foundation becomes a strategic asset that accelerates your growth. ![Data Architecture Diagram](https://rittmananalytics.com/lovable-uploads/a30311ba-53d8-4663-8700-574d299bc15f.png) ## It Starts With A Strategic Discovery Sprint Before a single line of code is written, we partner with you in a comprehensive Discovery and Strategy Sprint. This ensures that every technical decision is directly tied to a business outcome and that the platform we build is precisely what you need to succeed. STEP 1 ### Data Strategy Framework We define your strategic goals and the core business questions you need to answer. Our proven frameworks streamline this process from days to hours, establishing a clear value proposition and measurable success metrics for the entire project. STEP 2 ### Discovery of Requirements & Business Problems We are experts at translating your high-level needs into concrete, evidence-based technical requirements. We use a scientific, repeatable process to map your business data solutions, ensuring complete alignment between our teams. STEP 3 ### Design and Data Architecture We design an optimal information architecture and data model that maximizes the performance, cost-effectiveness and consistency of your modern data stack, ensuring high data quality and discoverability from day one. STEP 4 ### Project Roadmap & Agile Delivery Plan With a clear strategy and architecture, we create a detailed, 3-6 month delivery roadmap. We work in agile sprints, delivering continuous value and providing you with a clear timeline and focus for the implementation phase. [Learn More About Our Data Strategy Framework](https://rittmananalytics.com/services/data-strategy-framework) ## From Blueprint to Reality: Agile Platform Delivery The Discovery Sprint provides the strategic blueprint. We then move into a collaborative, agile delivery process to bring your data platform to life. Working in focused sprints, we typically tackle subject areas one by one, building out: ![Data pipeline diagram](https://rittmananalytics.com/lovable-uploads/67cbe393-8df0-4b67-bce7-cacff2fa96ed.png) ### Automated Data Pipelines Engineering solid and reliable pipelines to centralize your application, database and customer event data. ![Data modeling diagram](https://rittmananalytics.com/lovable-uploads/be85f08b-26a4-4525-8b7b-bdc83dd3f60f.png) ### Data Architecture & Data Modeling Translating the logical design into an optimized warehouse structure and data platform. ![Dashboard UI](https://rittmananalytics.com/lovable-uploads/77ed01d6-566a-4cfa-a674-0a3a684dba55.png) ### Dashboards, Reports & AI Creating the initial set of high-value dashboards and agentic workflows that deliver immediate insights. Engagement Models ## Three ways to work with us Defined scope, named senior team, no surprises. Every engagement starts with a discovery call. ### Discovery & strategy sprint 2-4 weeks · fixed price from £8k - Stack and data audit, stakeholder interviews - Target architecture and semantic model design - Costed roadmap you could run without us Most Common ### Platform build & delivery 3-6 months · agile sprints from £30k - Pipelines, warehouse, semantic layer, BI & AI - Working software every two weeks - Knowledge transfer built into every sprint ### Embedded team & enablement Monthly · flex up or down from £10k/mo - Senior consultants working inside your team - Shared Slack, stand-ups and sprint goals - Coaching that makes us progressively unnecessary These are typical figures. Every scope is priced after the discovery call, and we'll tell you if a smaller engagement would do. --- # Data Warehouse Migration - Modernise Your Legacy Data Platform | Rittman Analytics Canonical: https://rittmananalytics.com/services/data-warehouse-migration URL: https://rittmananalytics.com/services/data-warehouse-migration Description: A like-for-like legacy warehouse migration means months manually translating SQL, pipelines, syncs, and BI content. We use our AI-powered Wire Framework to automate this translation, delivering your modernized platform faster, cheaper, and more reliably. --- Migrating it is holding you hostage. We can help. A like-for-like legacy warehouse migration means months manually translating SQL, pipelines, syncs, and BI content. We use our AI-powered Wire Framework to automate this translation, delivering your modernized platform faster, cheaper, and more reliably. [Assess your migration](https://mark-9o3ijqpd.scoreapp.com/)[See client results ›](https://rittmananalytics.com/case-studies) Free. 15 questions. About 3 minutes. ## The Decision to Migrate Was Easy. The Execution is a Nightmare. You know you need to move to a modern data stack like BigQuery or Snowflake, but a like-for-like migration is notoriously slow and risky. To execute a flawless cutover without paralyzing your business, you need a partner with a fundamentally smarter blueprint. We are that partner. ### Your Challenges: #### The Infinite Timeline Manually translating SQL dialects, rerouting ingestion pipelines, and rebuilding BI dashboards piece-by-piece takes months. Deadlines inevitably slip, budgets balloon, and stakeholder patience vanishes long before the finish line. #### Roadmap Paralysis A major migration forces you to pull your best engineers and analysts off value-generating work. Instead of delivering new strategic insights or features, your smartest people are trapped doing repetitive, mechanical code translation. #### The Sunk-Cost Rollback Untangling deeply embedded business logic and downstream syncs gets so complex that momentum stalls. Teams get overwhelmed, often resulting in paying for two redundant platforms in parallel, or worse, abandoning the migration entirely. ### Our Solution: AI-Accelerated Platform Migration We use our AI-powered Wire Framework to automate the heavy lifting of translating your SQL, pipelines, and BI content. We deliver your fully modernized, like-for-like platform faster, cheaper, and with guaranteed reliability, ensuring a flawless cutover that keeps your internal team focused on building the business. #### Proof in Practice: See how Carwow accelerated its migration from Snowflake to Google BigQuery with Wire, translating around 1,700 dbt models in weeks while its data team kept the existing platform running. [Read More →](https://rittmananalytics.com/case-studies/carwow) ![Carwow vehicle presenter](https://rittmananalytics.com/__l5e/assets-v1/34616742-f8f1-4d30-8069-4bf503c0e5f1/carwow-image-hero.webp) Carwow ## Blueprint Your Cutover in Hours, Not Weeks Large-scale data platform migrations are less about writing SQL than managing risk. Thousands of models, connectors, orchestration workflows, semantic objects, dashboards and reverse ETL syncs all need to be tracked, migrated, regression tested and reassembled into a platform that behaves exactly like the one it replaces. Wire automatically deploys specialist AI-powered agents to discover and catalog all of your legacy platform components, identifying migration tasks and assessing their complexity, storing details in its migration inventory. Wire dispatches one agent per layer run concurrently each layer’s audit is independent of the others Audit agentsone per platform layerin parallel db-object-audit-generatedatabase objects every database and schema every table and view, with size and rows stored procedures and UDFs masking policies and tagged columns the full RBAC graph of roles and grants security-audit-generatethe same connection, different questionssecurity which roles hold which grants which columns carry PII tags Findings determine which IAM bindings and BigQuery policy tags target-setup creates. dbt auditdbt models assigns every model a complexity tier triviallowmediumhighblocked ingestion pipeline agentingestion pipelines details of every connection details of every pipeline reverse ETL agentreverse ETL details of every sync details of every destination …and one agent for every further layereach independent until results are assembled **MCP**  legacy DW server · account · role · warehouse Legacy DW platform the system being migrated from, typically thousands of objects **Databases & schemas** **Tables & views** size and row count **Stored procedures & UDFs** **Masking policies** and their tagged columns **Roles & grants** the full RBAC graph MCP Ingestion pipeline platform connectionspipelines MCP Reverse ETL platform syncsdestinations Migration inventory a single file, a tabular record of every item, one row per object, typically thousands pendingtranslateddeferredmigratedfailed objecttypesize · rowsstatecomplexity tablemigrated— UDFpending— dbt modeldeferredmedium pipelinetranslated— syncfailed— ⋮  one row per object dispatch back-and-forth over MCP assemble 1 2 3 ## Evidence, Not Assurances We eliminate the guesswork from re-platforming. Our process ensures zero silent errors through rigorous, step-by-step validation: - Full Dependency Mapping: Complete snapshot of your existing architecture. - Deep Translation: Precise dialect translation handling complex edge cases (e.g., type coercion, division by zero). - Data Reconciliation: Automated auditing of row counts and value totals in a secure sandbox prior to PR. - Transparent Handover: Clean, manageable PRs backed by an auditable register of differences. Consultancy delivery workspace · stages 1-6 Client repository & CI The only path into production code, client-reviewed, client-merged. Target production warehouse Only the client's scheduler ever writes here. Development sandbox isolated project on the target warehouse Migration inventory Tabular record of every item to be migrated, drives waves, sequencing and daily status. Item · type Wave orders\_daily dbt model2 pii\_row\_mask security rule1 crm\_contacts reverse ETL sync3 events\_load ingestion pipeline1 … and the rest of the estate 1Snapshot & audit The client's **legacy dbt package**, written in the legacy dialect, is mirrored into the delivery workspace. An audit maps every model's features and dependencies, and batches the estate into **waves**. 2Translation SQL and configuration rewritten to the target dialect under the agreed **translation guide**, explicit casts on joins, safe division, JSON handling, materialisation rules. Translated files live in the workspace, not the client repo. 3Validation & lint gate Every translated model must parse and compile in the target dialect, and pass the engagement's lint rules. **Nothing proceeds on translation alone.** 4Development build Built into a scratch dataset in the sandbox. A **cost screen** runs first: an estimated-scan check holds unusually expensive builds for authorisation. Proves the model runs against real data without touching production. deferred inputs, reads real production tables, **read-only** 5Equivalence test Compares the **migrated package's sandbox output** with the **source warehouse's contents**, as much data as the sandbox holds: row counts, window endpoints, sums, distinct counts. Four exits **Exact pass**, proceed to release **Qualified pass**, read-timing only, proceed **Availability gap**, data request; model waits **Defect**, back to translation, re-run 3-5 6Release assembly Passing models are grouped onto a branch of the client's repo. Four gates before release: 1Whole project parses with the new models in place 2Local replica of the client's CI passes 3Branch rebuild against the production graph, sandbox writes only 4Branch vs legacy re-check on every model A defect here pulls the model from the branch. External feeds & exports release on exact verdicts only 7Review & merge The client reviews the pull request and their CI runs independently. Merge places the model in the production codebase. 8Production build The client's scheduler builds the merged models into production on its normal cadence. 9Post-merge verification Compares the **production model's output in the target warehouse** with the same model's output in **legacy production**. Internal models **ship then verify**, fixes go forward via a new pull request. Verdicts land in the **tracking register** → daily status reporting. Numbered data requests Availability gaps, source history missing on the target, become numbered requests to the client. The model **waits**; resolution re-runs stages 4-5. Gaps found anywhere in the flow land on this list. Legacy dbt packagethe client's existing model code Legacy warehouseread-only throughout × many models in parallel pass pull request client CI · merge contents mirrored in **Single pooled runner**, every legacy read serialises through one connection legacy DW contents sandbox output branch re-check legacy production output target production output defect, fix the translation, re-run 3-5 fix forward, post-merge differences return as a new pull request availability gap, model waits resolved, re-run 4-5 ## Machine Speed. Human Judgment. We don't throw armies of junior consultants at a migration. We deploy an orchestrated swarm of AI agents, executing around the clock, strictly governed by a senior human director. The Orchestrator: A central AI translates human strategy into parallel workflows, resolving dependencies and routing data models to where they need to go next. Specialized Swarms: Dedicated agent lanes work 24/7 with hard safety limits. They specialize entirely on discrete tasks: translating SQL dialects, building sandboxes, debugging data discrepancies, or assembling release branches. Automated Rework: Errors don't wait for a sprint review. Defective models are instantly routed back for translation rework, then gated and re-tested until row counts and outputs match the legacy source perfectly. Absolute Transparency: Machines do the mechanical heavy lifting; humans make the commercial and strategic judgment calls. Every agent action is audited, checked against real data, and logged in a transparent register before your team reviews a single line of code. Tier 01 Release director One human Client communications Rulings and waivers Approval gates Judgment catches Fleet-size and budget decisions Rulings, waivers and approval gates Judgment catches Tier 02 Orchestrating session One Fable-class session Dispatches and monitors lanes Owns the register (single writer) Consolidation and backstop passes Assembles PR evidence Amends process docs the day a ruling lands Dispatch Monitoring and PR evidence Lane 01 Translate wave models Lane 02 Validate and lint Lane 03 Build playground copies Lane 04 Run equivalence battery Lane 05 Verify post-merge Lane 06 Drill count divergence Lane 07 Rebuild sync audit Lane 08 Check branch copies Tier 03 Lane agents 6–12 concurrent Stage-scoped work: translation, validation and evidence, one stage at a time. Selected Work ## AI-Powered Data Migration in-Practice 0 TB of data migrated and verified in production 0 dbt models translated from legacy to target SQL dialect 0 days to complete the platform migration --- # Data Strategy Framework - Transform Business Needs into Actionable Roadmaps | Rittman Analytics Canonical: https://rittmananalytics.com/services/data-strategy-framework URL: https://rittmananalytics.com/services/data-strategy-framework Description: Transform your business needs into actionable data roadmaps with our proven data strategy framework. Clear vision, prioritized goals, scalable solutions. --- Your Strategy Isn't. ## Let's build your blueprint for success. A successful data platform isn't built on technology alone; it's built on a clear, well-defined strategy. Our Data Strategy Framework is an intensive, fixed-scope engagement designed to transform your business needs into a prioritized, actionable, and realistic delivery roadmap. ## When Ambition Meets Ambiguity Embarking on a data analytics project can unlock valuable insights, drive business growth, and enhance decision-making processes. However, the path to achieving these outcomes is often fraught with challenges. ## Your Challenges: ### "You Don't Know Where to Start" You have a dozen competing priorities from different stakeholders. You know you need to modernize, but you lack a clear, unified vision of what "good" looks like or a framework for making strategic trade-offs. ### "Your Team Lacks a Clear Mandate" Your talented data team is stuck. Without a clear set of business goals and a prioritized roadmap, they can't effectively design the right data models or architecture, leading to analysis paralysis and wasted effort. ### "You Fear a Costly Misstep" You understand that a data platform is a significant investment. You can't afford to spend months building a solution based on guesswork, only to find it doesn't solve the core business problem or scale for the future. ## Our Solution: Data Strategy Framework Our Data Analytics Strategy Framework aims to provide long-term cost and time savings through an upfront investment in discovery, design and planning, ensuring that project goals are met through a  realistic and deliverable project roadmap. #### Proof in Practice: Read how we worked with Conversion Interactive Agency to define a data strategy and delivery roadmap to to increase their team's analytics capabilities and implement embedded analytics within their LeadLink 2.0 product. ![Conversion Interactive case study](https://rittmananalytics.com/lovable-uploads/0d75a221-3dff-4ece-9bb5-f3fed6f1b84a.png) ## Our Data Strategy Framework: A Four-Stage Blueprint for Success A structured path from chaos to clarity ![Strategy and business goals framework](https://rittmananalytics.com/lovable-uploads/86c0f273-5421-4f71-867c-4a2bb84fdcce.png) ### Stage 1: Define Strategy & Business Goals We start by identifying and articulating the core business outcomes your data platform must support. Using our proven methodologies, we facilitate streamlined workshops to establish clear, measurable success metrics and a value proposition for the entire initiative. ![Capability needs assessment](https://rittmananalytics.com/lovable-uploads/6a36d080-7307-44e7-890c-6575d27a4eb8.png) ### Stage 2: Discover Requirements & Problems-to-be-Solved We are experts at translating your high-level needs into concrete, evidence-based technical requirements. We engage with your key stakeholders to map your business challenges to specific data solutions, ensuring complete alignment. ![Data architecture design](https://rittmananalytics.com/lovable-uploads/419294ae-0109-4b9a-af90-f6690b162871.png) ### Stage 3: Design a Future-Proof Data Architecture A data function is only as good as its foundation. We design an optimal information architecture and logical data model that improves performance, scalability, data quality, and discoverability, ensuring the platform you build is ready for the future. ![Executive summary and roadmap](https://rittmananalytics.com/lovable-uploads/5df42187-107c-42a1-bffb-f75c4712a1f1.png) ### Stage 4: Create the Roadmap & Agile Delivery Plan We consolidate all findings into a concise executive summary and a detailed 3-6 month delivery roadmap. This provides a clear, prioritized, and realistic plan for implementation, giving you the confidence to move forward. ## Ready to Build Your Data Strategy with Confidence? Stop the analysis paralysis and get a clear, actionable blueprint for your data platform. Schedule a free discovery call to learn how our Data Strategy Framework can set you up for long-term success. --- # Data Team Enablement & Transformation - Scale Your Analytics Capabilities | Rittman Analytics Canonical: https://rittmananalytics.com/services/data-team-enablement URL: https://rittmananalytics.com/services/data-team-enablement Description: Transform your data team from reactive to strategic. Our enablement programs help analytics professionals master modern tools, build better processes, and deliver high-impact insights that drive business growth. --- Your Data Platform Didn't. Let's Fix That. We partner with successful organisations to transform their overwhelmed data teams from a reactive cost centre into a proactive, strategic asset. We deliver more than technology; we deliver the organisational change that makes it work. [See client results ›](https://rittmananalytics.com/case-studies) 45 minutes · no obligation · you'll speak with a founder, not sales Trusted by industry leaders like... [![ROUVY logo](https://rittmananalytics.com/lovable-uploads/bf807715-48cb-49e0-8424-9e6174a03664.png)](https://rittmananalytics.com/case-studies/rouvy)[![Pleo logo](https://rittmananalytics.com/lovable-uploads/b6fa83f2-d563-4737-8c54-6bcc6c379ed3.png)](https://rittmananalytics.com/case-studies/pleo)[![Ageras Group logo](https://rittmananalytics.com/lovable-uploads/ec80f8d6-7b91-43e0-bd0e-c5cced02ced4.png)](https://rittmananalytics.com/case-studies/ageras-group) ## For Data Leaders: Turning Your Team into a Strategic Force You've invested in a data team, but you're not seeing the ROI you expected. Your team is talented but siloed, reactive, and struggling to keep up with the demands of the business. ### Your Challenges: #### A "Flat" or "Directionless" Team Your data team is stuck in a cycle of maintenance and firefighting, leading to low morale and a lack of innovation. #### No Single Version of the Truth Your data warehouse has grown organically, leading to multiple sources of truth, inconsistent metrics, and a deep-seated distrust in the data. #### BI Tools That Are Anything but "Self-Service" Your BI tool (like Looker) is underutilized and locked down, forcing analysts to use risky workarounds and creating a " IT" problem. ### Our Solution: Strategic Data Team Enablement We partner with you to transform your team. We provide the embedded leadership, mentorship and proven processes to establish clear ownership, build a reliable data foundation, and empower your analysts to deliver proactive insights. #### Diagnose your Challenges Take our diagnostic assessment and receive a custom playbook with targeted advice for your specific challenges ![Diagnostic assessment illustration](https://rittmananalytics.com/lovable-uploads/a971585a-944d-4333-98fb-18c88253a45a.png) ## Our Proven Approach From Diagnosis To Sustainable Transformation A new data tool or a technical roadmap is not a strategy. Lasting data maturity comes from a disciplined process that aligns technology with the people who use it. Our three-step methodology is designed to do exactly that, moving your organisation from a state of data chaos to one of clarity, trust, and high performance. ### STEP 1: CONFRONTING HARD TRUTHS #### The Diagnostic Deep Dive Through in-depth interviews, observation of team rituals, and a rigorous analysis of existing documentation, we identify the root causes of your data challenges, be they technical debt, organisational misalignment, or cultural fatigue. Out of this discovery exercise, you get an unvarnished, holistic assessment of your current state. We present you with the foundational truths, allowing us to build a strategy that solves the right problems from the start. ### STEP 2: CREATING THE CHANGE #### Architecting a Scalable Foundation With a clear diagnosis, we architect a modern, scalable data platform tailored to your specific needs. This is where our deep technical expertise translates strategy into a functioning, reliable system. The result is a comprehensive, well-documented technical blueprint for success. You get a platform that is not only powerful but also maintainable, scalable, and designed to eliminate the inefficiencies that held you back. ### STEP 3: MANAGING THE CHANGE #### Embedding a Culture of Ownership This is our key differentiator. A perfect platform will fail if no one uses it correctly. We actively manage the human side of the transformation to ensure new processes are not just adopted but embraced. Through targeted training, ongoing mentorship, and clear ownership structures, you develop effective team rituals, and champion initiatives that drive a culture of clear ownership and accountability, leaving you with a team that has the skills, processes and a clear sense of purpose to drive value for years to come. ## From Vision to Reality: How We Helped Pleo Build a Modern Data Foundation Pleo's remarkable success and hyper-growth created a new challenge: the data infrastructure that had served them well during their start-up phase needed to evolve to match their future ambitions. Our discovery revealed a central data team navigating a period of change, deeply committed to maintaining core infrastructure, but in-need of support for fast business growth. > "Working with Rittman Analytics has been great. They have been a very helpful extension of our own Data team, rolling up their sleeves and building with us. That partnership was key to leveling-up our data foundations, which lets our team focus on what they do best: using data to drive innovation faster." Pri Nagashima, VP of Data, Analytics and AI at Pleo ### Our Partnership Delivered: - Accelerated delivery of Pleo's new analytics warehouse, integrating critical data modeling, orchestration and business intelligence capabilities - Increased capabilities for Pleo's internal data team through targeted training and knowledge sharing - Improved collaboration and trust between Pleo's central and distributed data teams, ensuring data outputs were well-aligned with the company's evolving business needs - A more solid, scalable data platform to support Pleo's continued growth ![Pleo mobile app interface showing modern data-driven features](https://rittmananalytics.com/lovable-uploads/c14d9362-202e-4854-9b16-8a17f1f6e283.png) [Read the Full Pleo Case Study →](https://rittmananalytics.com/case-studies/pleo) ## Ready to Transform Your Data Team? Let's discuss how our proven methodology can turn your data team into a strategic asset that drives real business value. --- # Embedded Analytics Solutions - Transform Your Product with Data | Rittman Analytics Canonical: https://rittmananalytics.com/services/embedded-analytics URL: https://rittmananalytics.com/services/embedded-analytics Description: Turn your product into a competitive advantage with embedded analytics. We help SaaS and product companies build powerful, white-labeled analytics experiences that drive user engagement and retention. --- Start Shipping a World-Class Analytics Feature. Your customers are demanding more than just data exports. They want actionable, embedded insights inside your application. But what starts as "adding a few charts" quickly spirals into a full-blown, resource-draining engineering project. We partner with SaaS companies to build secure, scalable, and high-impact customer-facing analytics. We provide the architecture, pre-built models, and implementation expertise to help you deliver a market-leading analytics experience, faster. ## Building In-App Analytics is Harder Than It Looks. SaaS teams often underestimate the complexity of delivering embedded analytics. Whether you're building a specific feature or implementing a framework, you face a common set of pitfalls that waste time, burn budget, and damage customer trust. ### Challenge 1: Adding a Complex Analytical Feature (e.g., Marketing Attribution) You think you're just adding a dashboard, but you're actually building a complex data modeling engine from scratch. Without deep expertise, this often leads to: - Brittle, oversimplified models that fail to meet customer expectations. - Noisy, untrustworthy results due to a poor data foundation. - Months of engineering "plumbing" instead of delivering value. - Features that don't scale and break under increased data volume. ### Challenge 2: Implementing a Framework (e.g., Cube.js) You've chosen a powerful framework like Cube.js, but you're discovering that implementation is fraught with risk. DIY attempts frequently result in: - Catastrophic security holes from poorly handled multi-tenancy. - Slow, unreliable dashboards that frustrate customers due to a lack of performance tuning. - Unnecessary reinvention of caching, access control, and BI layers. - A fragile, misaligned architecture that creates a disconnect between your data and product teams. In both scenarios, the outcome is the same: a high-cost project that fails to deliver strategic value and erodes customer trust. ## Deliver High-Impact Analytics, Without the Headaches. We bring both the strategic product thinking and the deep technical expertise to turn embedded analytics into a scalable, secure and valuable feature, not a liability. Our approach ensures you get it right. ### Secure, Scalable And Seamlessly Integrated By Design ### Embedded Analytics Is Only As Good As Its Implementation ### Go From Roadmap To Revenue Fast With Pre-Built Analytics ### Secure, Scalable And Seamlessly Integrated By Design #### Architect a Secure, Multi-Tenant Data Model from Day One. The biggest risk in embedded analytics is data leakage. We design and implement a solid, production-grade security model using best practices for authentication and row-level security, ensuring one customer's data is never exposed to another. #### Design for Performance and Scale : Don't let slow dashboards frustrate your customers. We design and tune high-performance caching and pre-aggregation within Cube, Embeddable and your framework that delivers insights in seconds, not minutes. By optimising how the framework queries your data warehouse, we prevent bottlenecks and ensure a fast, reliable user experience. #### Integrate Seamlessly with Your Existing development Workflows. Your analytics feature shouldn't be another data silo. We ensure your embedded analytics framework integrates directly with your existing dbt models and CI/CD pipelines, creating a unified development workflow with automated testing and validation and preventing fragile deployments. ![Secure, Scalable And Seamlessly Integrated By Design](https://rittmananalytics.com/lovable-uploads/14fac9a1-ffb4-4119-b16f-0c5a0a6d6d07.png) ## From Vision to Reality: How We Helped Insighta Build a Multi-Tenant Analytics Platform Insighta needed to build a scalable, multi-tenant marketing attribution application for their diverse client base. They knew that achieving their ambitious goals required external expertise in data engineering and analytics enablement. > "With a background in building multi-tenant solutions for clients in the marketing technology industry, Rittman Analytics were the ideal partner to help Insighta make our product vision a reality." Matthiew Liu, Co-Founder, Insighta ### Our Partnership Delivered: - A Scalable Multi-Tenancy Architecture using dbt and Dagster. - Customized Conforming Data Layers to integrate disparate data sources like Shopify and NetSuite. - Predictive Analytics & ML Workflows to deliver actionable insights. - Rapid Deployment and Automation to accelerate the onboarding of new clients. ![Insighta case study](https://rittmananalytics.com/lovable-uploads/e3fe4eae-383f-4794-bce5-ebca44544b32.png) [Read the Full Insighta Case Study →](https://rittmananalytics.com/case-studies/insighta) ## Ready to Build an Analytics Experience Your Customers Will Love? Avoid the costly pitfalls of DIY development. Let's have a strategic conversation about how our expertise can help you deliver a secure, scalable, and high-impact analytics feature that drives adoption and proves value. --- # Technical Consulting - Expert Data & Analytics Consulting | Rittman Analytics Canonical: https://rittmananalytics.com/services/technical-consulting URL: https://rittmananalytics.com/services/technical-consulting Description: Expert technical consulting for data platforms, analytics infrastructure, and modern data stack implementation. Solve complex data challenges with proven expertise. --- Most Pressing Data Challenges. You have a clear objective. You know the tools you want to use. Now you need a senior, hands-on partner with deep technical expertise to execute with precision and speed. Our Expert Services provide high-impact implementation and consulting for dbt, Looker, and Generative AI. We are the trusted partner for technical teams who need to get it right, the first time. In partnership with... ![Snowcat Cloud](https://rittmananalytics.com/lovable-uploads/b414e4f8-ce94-4f89-b4d5-bcb73fe3f545.png)![Snowflake](https://rittmananalytics.com/lovable-uploads/0d930e52-78f6-457e-ae69-043f540f62f4.png)![Google Cloud](https://rittmananalytics.com/lovable-uploads/e481a93b-7462-46db-9a41-98bae82c401b.png)![Cube](https://rittmananalytics.com/lovable-uploads/544f7c86-c2a1-495c-b256-cf90a9c327ae.png)![Looker](https://rittmananalytics.com/lovable-uploads/eec78f99-61e4-4eb3-9d7b-9d5bb4845de8.png)![dbt Labs](https://rittmananalytics.com/lovable-uploads/093f128d-bace-465a-8cd6-acc2d870f13d.png)![Dagster](https://rittmananalytics.com/lovable-uploads/968ca976-3824-469d-9320-3728a0488312.png)![Embeddable](https://rittmananalytics.com/lovable-uploads/1a278600-bf96-465b-b9d0-c126bae2b885.png)![Hightouch](https://rittmananalytics.com/lovable-uploads/a38c5f3f-5b44-470e-8bac-a435ae7c3fbb.png)![RudderStack](https://rittmananalytics.com/lovable-uploads/18ce20e2-af5d-46b1-a6cb-2350cc8f1aac.png)![Fivetran](https://rittmananalytics.com/lovable-uploads/d50cfde2-3610-4afd-ab0d-1dac8debad57.png) ## Consulting Packages and Tools for the Modern Data Stack Accelerate your delivery and time-to-value with our specialized packages and open-source tools for dbt, Looker, and Generative AI ![dbt workflow](https://rittmananalytics.com/lovable-uploads/b03348c3-999a-4fcc-a59b-4cd871e83de2.png) ### dbt Consulting & Implementation From initial setup to complex refactoring, we help you master dbt. We establish best practices for testing, documentation, and CI/CD, and help you build scalable, maintainable data models. [Learn More](https://rittmananalytics.com/partners/dbt) ![Looker dashboard](https://rittmananalytics.com/lovable-uploads/c03ae80d-f61b-4fcb-a19c-8db195fce4d5.png) ### Looker Development & Optimization We help you get far more out of your Looker instance. We design intuitive Explores, build solid LookML models, and implement governance frameworks that enable true self-service analytics at scale. [Learn More](https://rittmananalytics.com/partners/looker) ![AI Analytics Assistant](https://rittmananalytics.com/lovable-uploads/331eaec2-8f6b-4edb-ad28-2a429e3a3b3b.png) ### Generative AI & Advanced Analytics Go beyond traditional dashboards. We leverage Google Cloud's Vertex AI and frameworks like LangChain to build modern AI-powered features, from conversational chatbots to automated pre-analysis and insight generation. [Learn More](https://rittmananalytics.com/partners/google-cloud) ![Cube Cloud interface](https://rittmananalytics.com/lovable-uploads/4c0fe9f5-ae0d-4593-bb1d-461181722914.png) ### Cube Development & Implementation We help you build powerful semantic layers with Cube. From API design to advanced caching strategies, we enable you to create performant, scalable data APIs that power modern analytics applications. [Learn More](https://rittmananalytics.com/partners/cube) ![RudderStack connections](https://rittmananalytics.com/lovable-uploads/1eea156a-d76e-41c5-a639-d726c7be1e52.png) ### RudderStack Data Pipeline Engineering We help you implement solid customer data infrastructure with RudderStack. From event tracking to warehouse syncing, we build reliable data pipelines that scale with your business. [Learn More](https://rittmananalytics.com/partners/rudderstack) ![BigQuery data pipeline](https://rittmananalytics.com/lovable-uploads/9f7bdffa-9cba-4e43-b9ad-bafdf3410242.png) ### Google BigQuery Optimization We help you maximize the potential of Google BigQuery. From query optimization to cost management, we implement best practices that improve performance while reducing expenses. [Learn More](https://rittmananalytics.com/partners/google-cloud-looker) ## A Partner That Understands Your World We are engineers and builders at our core. We speak your language and understand the nuances of building solid, maintainable data systems. We partner with your team to not just deliver a solution, but to leave you with cleaner code, better processes, and elevated capabilities. ![Company logo](https://rittmananalytics.com/lovable-uploads/528a6b69-96ee-4950-846a-d2047fd7ec76.png) > "Rittman Analytics' deep technical expertise with the modern data stack increased our data team's capabilities." ![Kelley Walkup](https://rittmananalytics.com/lovable-uploads/1ffd4860-b788-484c-8dc8-dc9420ab5966.png) Kelley Walkup President and CEO, Conversion Interactive Agency ![Company logo](https://rittmananalytics.com/lovable-uploads/f5ed2515-9655-42d1-8b04-d600d16ee47f.png) > "Rittman Analytics was instrumental in designing the architecture of the new data stack and providing senior engineering support." ![Erik Hegely](https://rittmananalytics.com/lovable-uploads/f6935b31-b28f-4498-936d-5537787250a7.png) Erik Hegely Growth Director, ROUVY ## Let's Solve Your Toughest Data Problem. Whether you have a well-defined project or need an expert partner to help scope a technical solution, we're here to help. Schedule a free, no-obligation call to discuss your specific needs with one of our senior engineers. --- # Wire Framework, AI & Agentic Analytics Delivery Blueprints | Rittman Analytics Canonical: https://rittmananalytics.com/services/wire-framework URL: https://rittmananalytics.com/services/wire-framework Description: The Wire Framework is Rittman Analytics' opinionated, production-ready dbt and Dagster blueprint for building modern, AI-ready data platforms, faster, with fewer mistakes. --- New · v3.4.20 AI-accelerated delivery, from SOW to dashboards. # Build data platforms in days, not weeks. Wire is the delivery framework that turns Claude Code and Gemini CLI into a senior analytics engineer that already knows your conventions. Every artifact — from requirements to LookML — generated, validated and reviewed against the same playbook we’ve refined over 20+ years. [Read the Docs](https://docs.rittmananalytics.com/) [Read our Case Study](https://rittmananalytics.com/case-studies/barton-peveril) 10 Release types 15 Artifact types ~25min Autopilot SOW → run acme-platform · 01-data-foundation $ /wire:requirements-generate 01-data-foundation → reading engagement/context.md + sow.md → applying spec workflows/requirements/generate.md ✓ generated requirements\_specification.md · 13 sections $ /wire:requirements-validate 01-data-foundation ✓ 13/13 checks passed $ /wire:requirements-review 01-data-foundation ⌛ awaiting sponsor approval requirements conceptual\_model data\_model dbt semantic\_layer dashboards Built on the stack you already trust ![Claude](https://rittmananalytics.com/wire-framework/assets/logo-claude.png) ![Gemini](https://rittmananalytics.com/wire-framework/assets/logo-gemini.png) ![dbt](https://rittmananalytics.com/wire-framework/assets/logo-dbt.png) ![BigQuery](https://rittmananalytics.com/wire-framework/assets/logo-bigquery.jpg) ![Looker](https://rittmananalytics.com/wire-framework/assets/logo-looker.png) ![Fivetran](https://rittmananalytics.com/wire-framework/assets/logo-fivetran.webp) ![Dagster](https://rittmananalytics.com/wire-framework/assets/logo-dagster.svg) The challenge ## The Data Team's Dilemma: Speed vs. Trust Traditional analytics delivery is broken. Business leaders want answers yesterday, but the process is slow and fraught with risk. ### Your Challenges: - **Moving too fast** leads to inconsistent metrics, broken dashboards, and a gradual erosion of trust from the very stakeholders you're trying to serve. - **Moving too slowly** means missed opportunities, a frustrated business, and a data team that is perpetually seen as a bottleneck, not a strategic partner. - **AI is not a magic bullet**, used incorrectly, it just creates higher volumes of low-quality work. Used strategically, it can transform your team's entire operating model. ### Our Solution: AI-Augmented Process, Human-Led Judgment Our approach uses AI to automate repetitive tasks and accelerate feedback loops, freeing your team to focus on what matters most: complex business logic and stakeholder collaboration. #### Proof in Practice: See how we partnered with Barton Peveril College to deliver an AI-accelerated MVP analytics platform in just 15 working days, pioneering a new approach for UK education. [Read More →](https://rittmananalytics.com/case-studies/barton-peveril) ![Barton Peveril College students](https://rittmananalytics.com/lovable-uploads/barton-peveril-science-lab.png) Barton Peveril The problem we’re solving ## AI code generation gets the SQL right. It gets the methodology *wrong*. Drop an AI into a dbt project with no scaffolding and you get models that compile, then naming chaos, missing tests, drifting grain, all the things a senior analytics engineer wouldn’t miss. Without Wire ### Improvisation at every step The model knows the conventions. It just isn’t held to them. The result is technically valid SQL that, over 15 models, drifts in ways that erode the value proposition entirely. - Inconsistent naming across `stg_`, `int_`, `wh_` layers - Missing surrogate-key patterns and grain management - FK relationship tests skipped on half the models - Requirements untraceable to warehouse columns - Each engagement is a one-off with no compounding With Wire ### Generation, constrained. Each command reads its workflow specification before generating. The spec dictates upstream inputs, naming, structure, validation checks, and how state is updated. Every artifact looks like the team has been on the project for months. - Encoded conventions for dbt, LookML, pipelines and docs - Automatic prerequisite enforcement between artifacts - Validation is mechanical: pass / fail with named failures - Research findings persist across sessions and releases - Every engagement compounds onto the same framework Release types ## Ten shapes of engagement. One framework. Discovery to scope it. Ten delivery types for everything from a one-week dbt sprint to a full platform build. Agentic Commerce for AI-powered storefronts. Pick the type; the framework instantiates the right artifact queue. #### Discovery Shape Up planning: problem → pitch → release brief → sprint plan. 1–2 wk**4 artifacts** #### Full Platform SOW → production dashboards + trained users. End-to-end. 2–3 wk**15 artifacts** #### Dashboard-First Interactive mocks drive the data model; seed data enables immediate dbt. 1–2 wk**14 artifacts** #### Pipeline + dbt New pipeline plus dbt transformation layer; BI out of scope. 1–2 wk**7 artifacts** #### dbt Development Analytics engineering on existing pipeline infrastructure. 1 wk**5 artifacts** #### Dashboard Extension New dashboards on top of an existing, populated semantic layer. 3–5 d**4 artifacts** #### Enablement Training and documentation for an existing platform. 2–3 d**2 artifacts** #### Agentic Commerce AI-powered ecommerce: Lovable + 9 AI features (search, try-on, assistant). 1–4 wk**9 artifacts** #### Data Migration Lift, validate and cut over legacy warehouses, pipelines and dashboards with rollback coverage. 1–3 wk**8 artifacts** #### Platform Health Check Rapid audit of dbt, pipelines and dashboards with a prioritised backlog of fixes and quick wins. 2–3 d**3 artifacts** How it works ## How AI-Augmented Delivery Works Our methodology embeds AI into every stage of the analytics lifecycle, while keeping your expert team in complete control. ![AI-generated dashboard mockup](https://rittmananalytics.com/lovable-uploads/ai-accelerated-prototyping.png) ### Accelerated Prototyping Use AI to generate initial data models and dashboard mockups directly from your requirements, making stakeholder conversations concrete and productive from day one. ![Governance and code review process](https://rittmananalytics.com/lovable-uploads/ai-governance-speed.png) ### Governance at Speed Ensure every AI-generated asset passes through clear, human-led review gates. We maintain 100% explainability, semantic clarity, and trust in your final data products. ![Accelerated AI delivery timeline](https://rittmananalytics.com/lovable-uploads/ai-faster-iteration.png) ### Faster Iteration Instantly generate boilerplate code, documentation, and tests, allowing your engineers to focus on solving unique business problems, not on repetitive, low-value tasks. ![Collaborative review and validation](https://rittmananalytics.com/lovable-uploads/ai-deeper-collaboration.png) ### Deeper Collaboration Free your team from waiting on slow builds and manual processes. More time is spent with stakeholders validating insights and less time debugging pipelines. Why Wire ## Three gates, every artifact. Every artifact follows the same three-gate pattern. Substitute `` with `requirements`, `conceptual_model`, `dbt`, anything in the queue. The next gate doesn’t fire until the previous one passes, phase discipline is structural, not a checklist someone might skip. Gate 1 ### Generate AI reads the spec and upstream inputs, produces the artifact. Files land in the release folder. Gate 2 ### Validate Automated checks against conventions. **PASS** or **FAIL**, with the specific failures listed. Gate 3 ### Review You or the stakeholder approves. Status moves to `approved` or `changes_requested`. Each gate updates `status.md` and appends to `execution_log.md`. Nothing happens off the books, so a new team member picks up an engagement and is productive within an hour. VS Code extension ## Wire, right inside your editor. The Wire VS Code extension surfaces every `/wire:*` command, every artifact gate state, and the running execution log in the editor you already live in. It shares state with the CLI, so you can switch between them mid-engagement without a beat. ##### Status sidebar Every artifact’s gate state visible in a dedicated side-bar pane, one click to open the file. ##### Command palette `⌘⇧P` → type “Wire”. All 67 commands one keystroke away, with the release IDs auto-completed. ##### Inline terminal output Generate, validate and review run inside the editor terminal, with syntax-highlighted validation reports. ##### Live status.md The release’s `status.md` updates in real time as you run commands. Diff-friendly, git-friendly, review-friendly. Install Search *“Wire Framework”* in the VS Code Marketplace, or run `code --install-extension rittmananalytics.wire`. Requires VS Code 1.98+ and Claude Code. status.md · acme-platform Wire · 01-data-foundation requirements approved conceptual\_model approved data\_model running pipeline queued dbt semantic\_layer dashboards Files status.md execution\_log.md 1\--- 2release\_id: "01-data-foundation" 3release\_type: "full\_platform" 4artifacts: 5 requirements: 6 generate: complete 7 validate: pass 8 review: approved 9 conceptual\_model: 10 generate: complete 11 validate: pass 12 review: approved 13 data\_model: 14 generate: in\_progress 15\--- 17\# Release status 18 19Day 3 of 15. Requirements + conceptual 20model signed off by Priya on Monday. Wire Autopilot ## Hand it the SOW. Walk away. For standard engagement shapes, Autopilot takes a Statement of Work and runs the entire lifecycle, discovery sprint, then every downstream delivery release the discovery identifies, autonomously. - **Eight context questions** upfront, client name, repo mode, tracker, document store. No release-type guessing. - **Self-validates and self-approves** internal artifacts, with up-to-three retries on validation and self-review. - **Safety gates** before any phase that touches external systems, pipeline activation, data refactor, deployment. - **Resumable from checkpoint**, interruptions cost minutes, not a re-run. #### autopilot · acme-platform.sow.pdf Phase 1 · Discovery sprint · in progress ✓ problem\_definition10 sections · auto-approved Complete ✓ pitchAppetite: 3 wk · 3 downstream releases identified Complete ↻ release\_briefValidate (attempt 1 of 3) Running · sprint\_planQueued Queued ! pipeline · safety gateWill pause before activating Fivetran Gate Worked example · Barton Peveril ### Live pastoral analytics, end-to-end, in 8 working days. Barton Peveril Sixth Form College needed a live pastoral-risk dashboard on top of ProSolution MIS and Focus, with safeguarding-grade data handling and same-week training for SPAs and pastoral leads. One `full_platform` release. 15 artifacts. SOW signed on Monday; dashboards live and end-users trained by the following Wednesday. [Read the case study](https://rittmananalytics.com/#) 8 d Working days · SOW to live 9 dbt models · 47 tests 100% PK + FK test coverage £7.1k Budget · 35 hours Built-in integrations ## Sits inside the tools your team already uses. Wire ships as a plugin for Claude Code and an extension for Gemini CLI. State is plain markdown in git. Integrations are MCP-native — one command to authenticate, then state syncs automatically. ![Fivetran](https://rittmananalytics.com/wire-framework/assets/logo-fivetran.webp)![Dagster](https://rittmananalytics.com/wire-framework/assets/logo-dagster.svg) Ingestion + orchestration Fivetran · Dagster Pipeline design + cost analysis. Connectors verified before generate; assets-first Dagster. Issue tracking Jira · Linear Bidirectional sync from `status.md` to your tracker. Fail-graceful — never blocks delivery. Document store Confluence · Notion Every approved artifact auto-publishes to your client’s space. Discovery docs reviewable in their tooling. Meetings Fathom MCP `utils-meeting-context` pulls transcripts into the engagement before every call. ## Start your next engagement on Wire. Book a 30-minute walk-through and see Wire run end-to-end on one of your SOWs, or one of ours. Take the recording away whether or not we work together. [Book a 30-min walk-through](https://rittmananalytics.com/#) [Read the docs](https://docs.rittmananalytics.com/) #### What you’ll see - SOW → live dbt + LookML in under 30 minutes via Autopilot. - The exact validation rules a Rittman senior would apply. - How the same playbook runs across BigQuery and Snowflake. - How to roll Wire into your own delivery team. --- # E-commerce & Retail Analytics Solutions - Drive Growth with Data | Rittman Analytics Canonical: https://rittmananalytics.com/solutions/ecommerce-retail URL: https://rittmananalytics.com/solutions/ecommerce-retail Description: Transform your e-commerce and retail business with unified analytics. We help retailers consolidate sales data, optimize customer journeys, and drive growth through data-driven insights across all channels. --- Your brand is growing fast, but your data is stuck in silos. You're trying to make critical trading decisions using a patchwork of disconnected reports from Shopify, NetSuite, Google Ads, and countless spreadsheets. We partner with high-growth e-commerce and retail brands to build a single source of truth for your entire business. We unify your sales, inventory, marketing, and operational data into one analytics platform, empowering you to make faster, more profitable decisions. The data partner for high-growth retail brands, including: [![RIXO](https://rittmananalytics.com/lovable-uploads/f08776bd-4826-4e99-970c-0a2aa82924ae.png)](https://rittmananalytics.com/case-studies/rixo)[![Popsa](https://rittmananalytics.com/lovable-uploads/0efdef7b-4690-4f31-8c43-0fa67b294f2f.png)](https://rittmananalytics.com/case-studies/popsa) ## When Disconnected Data Creates Costly Blind Spots As a retail leader, you're constantly balancing inventory, marketing spend, and customer experience. But without a unified data platform, you're facing a common set of challenges that directly impact your bottom line. ### You Have No Single View of Performance Your weekly trading review is a painful, manual process. Your team spends hours every Monday pulling data from dozens of sources into spreadsheets, leaving little time for actual analysis and strategic decision-making. ### You're Drowning in E-commerce Complexity Your data can't handle the specific nuances of retail. You struggle to link returns to original sales, account for gift card revenue correctly, or report accurately using the 4-5-4 retail calendar. ### You Can't Trust Your Inventory and Sales Data Your team spends hours every week manually compiling sales and stock levels from different warehouses, e-commerce platforms, and retail stores into spreadsheets. The process is slow, error-prone, and the final numbers are often disputed. ## A Single Source of Truth for Your Entire Operation We help you build a modern data foundation that connects every part of your business. Our solutions for Google Cloud are designed to deliver the clarity and confidence you need to make smarter, faster decisions. ### Unified Sales & Revenue Analytics We centralize your sales data from every channel, e-commerce, physical stores, and wholesale, into a single, reliable view. We implement the custom business logic needed to accurately track revenue, margins, returns, and customer splits. ### Multi-Touch Marketing Attribution Go beyond last-click and understand the true ROI of your marketing spend. Our pre-built attribution models connect your ad spend from Google and Meta directly to sales revenue, giving you a clear picture of what's working. ### Product & Inventory Optimization Accurately meet demand and optimize your product mix. We build the data models that allow you to analyze performance by vendor, category, and SKU, providing the insights needed for effective merchandising and inventory planning. ### Automated "Trade Pack" Reporting We replace your manual Monday reporting with automated dashboards that deliver complete weekly performance across online and in-store sales, advertising efficiency, brand development, and wholesale. ### Unified Customer Analytics We centralize your data to build a true 360-degree view of your customers. We connect purchase history with interaction behavior across all your touchpoints, allowing you to understand your best customers. ### Modern, Scalable Data Infrastructure We architect and build the solid data foundation you need to scale. From connecting to complex sources like NetSuite to building a dimensional data warehouse with dbt and Looker, we ensure your infrastructure is secure and reliable. ## Our Expertise in Action: Creating a Single View for RIXO ![RIXO](https://rittmananalytics.com/lovable-uploads/05cac939-3b12-4ce0-bbee-801c8d2ab1bb.png) ### Creating a Single View of Trading Operations for a Global Fashion Brand RIXO, the London-based fashion brand, had grown rapidly into an international business with e-commerce, physical retail, and wholesale channels. They needed a unified view of their data to monitor revenue streams, stock levels, and overall business performance, but lacked the in-house analytics capability to build it. > "Thank you for all your wonderful help on getting us up and running with Looker and keeping everyone on board along the way!" [Read the Full RIXO Case Study →](https://rittmananalytics.com/case-studies/rixo) #### Our Partnership Delivered: - A **centralized data warehouse** on Google BigQuery, integrating data from NetSuite, Shopify, Google Ads, and more. - A set of automated Looker dashboards to replace a time-consuming manual slide deck, freeing up a third of the head office staff from hours of weekly spreadsheet work. - A terrific foundation and a clear roadmap for their new data team, enabling them to function at a higher level and own their analytics future. --- # EduTech & Digital Learning Analytics Solutions | Rittman Analytics Canonical: https://rittmananalytics.com/solutions/edutech URL: https://rittmananalytics.com/solutions/edutech Description: Transform educational outcomes with advanced EduTech analytics. Link platform usage to student outcomes and drive institutional success. --- Demonstrable Impact. In the competitive EdTech landscape, success is measured by more than just platform usage. You need to prove that your product delivers real educational outcomes, drives institutional adoption, and supports sustainable commercial growth. We help EdTech companies build data platforms that connect the dots between learner engagement, content efficacy, and business performance. We unify your data to deliver the actionable insights you need to improve learning outcomes and accelerate growth. The data partner for innovative education companies, including: ![Welbee](https://rittmananalytics.com/lovable-uploads/a1713804-0140-4f34-97ba-670e3b230cc7.png)![BridgeU](https://rittmananalytics.com/lovable-uploads/968ca976-3824-469d-9320-3728a0488312.png)![Conversion Interactive](https://rittmananalytics.com/lovable-uploads/6fe9e326-335e-4023-9fa6-9869257b511d.png) ## When Your Data Doesn't Tell the Full Story Your platform generates a wealth of data on learner activity, but it's often siloed from the commercial and operational data that drives your business. This creates critical blind spots. ### You Can't Link Platform Usage to Outcomes You see that learners are active, but you can't easily measure how their engagement with specific content or features translates into better academic results, higher retention rates, or successful student journeys. ### You're Flying Blind on Institutional Adoption Your sales and CRM data is disconnected from your product usage data. You can't see which schools are truly engaged, which are at risk of churn, or which accounts represent the biggest expansion opportunities. ### You're Struggling to Provide Actionable Insights to Educators Your process for analyzing survey data or platform usage is manual and slow. This creates delays for schools and educators who need timely, data-driven recommendations to improve student wellbeing and performance. ## A Unified Platform for Data-Driven Student Growth We provide the strategic framework, technical implementation and and a single source of truth for your fast-growing EduTech or educational institution. ### Unified Learner & Customer Journey We centralize data from your platform, CRM, and marketing tools to map the entire journey, from initial prospect to engaged institution. We help you understand adoption patterns, identify at-risk accounts, and power your customer success teams with proactive insights. ### Outcome & Efficacy Measurement We build the data models that connect platform engagement with tangible outcomes. Whether it's attributing student success to specific interventions or measuring the impact of new features on retention, we help you prove the educational value of your product. ### AI-Powered Insights & Personalization Go beyond static dashboards. We leverage Generative AI and Text-to-SQL agents to create conversational analytics experiences. Empower educators and administrators to ask natural language questions and receive instant, actionable recommendations from their data. ### Modern, Scalable Data Infrastructure We architect and build the solid data foundation you need to scale. From implementing first-party event tracking with Snowplow to building a dimensional data warehouse with dbt and Looker, we ensure your infrastructure is secure, reliable, and ready for growth. ## Our Expertise in Action: Driving Innovation in EdTech We have a proven track record of helping EdTech leaders solve their most complex data challenges and build a competitive advantage through analytics. ![Client Logo](https://rittmananalytics.com/lovable-uploads/f843f2c5-4d4f-4232-ad64-e42d8e487715.png) ### Bringing AI-Driven Insights to Educator Wellbeing We partnered with Welbee to develop an AI-powered proof of concept for their "Intelligent Ed" platform, using Google's Gemini and LangChain to create a conversational chatbot that delivers instant, actionable insights from behavioral survey data. [Read Case Study →](https://rittmananalytics.com/case-studies/welbee) ![Client Logo](https://rittmananalytics.com/lovable-uploads/d6137955-b934-47ef-bee2-e5109bae3738.png) ### Measuring and Understanding User Engagement We worked with BridgeU to centralize their behavioral survey results and usage event data, building a dimensional data warehouse and Looker dashboards to help them understand which platform actions were most influential in moving students through the university application funnel. [Read Case Study →](https://rittmananalytics.com/case-studies/bridgeu) --- # FinTech Solutions - Rittman Analytics Canonical: https://rittmananalytics.com/solutions/fintech URL: https://rittmananalytics.com/solutions/fintech Description: Build secure, scalable data platforms for FinTech companies. Unify transaction data, ensure compliance, and accelerate growth with proven analytics solutions. --- In FinTech, data isn't just for dashboards, it's the foundation of your product, your compliance, and your customers' trust. A fragmented, unreliable data platform isn't just inefficient; it's a direct risk to your business. We partner with innovative FinTech and banking teams to build data platforms that deliver on the dual mandate of rigorous security and rapid growth. We unify your transaction, user behavior, and operational data into a single, reliable analytics layer that empowers you to innovate with confidence. The trusted data partner for innovative FinTech companies, including: ![Ageras Group](https://rittmananalytics.com/lovable-uploads/ec80f8d6-7b91-43e0-bd0e-c5cced02ced4.png)![Conversion Interactive](https://rittmananalytics.com/lovable-uploads/6fe9e326-335e-4023-9fa6-9869257b511d.png) ## When "Good Enough" Data Creates Unacceptable Risk Scaling a FinTech company means moving beyond siloed reporting and legacy systems. You're facing a unique set of data challenges where inconsistency isn't just an inconvenience, it's a liability. ### You Can't Get a Single View of the Customer Lifecycle Your data is fragmented across onboarding, KYC, payments, and support systems. This makes it impossible to accurately measure critical metrics like CAC and LTV, or to model risk effectively across the full customer journey. ### Your Data Lacks a Single Source of Truth Like many scaling companies, you have multiple dashboards created by different people from different sources. This leads to a lack of trust between stakeholders, delays in decision-making, and an inability to agree on core business definitions. ### You're Not Equipped for In-App, Real-Time Insights You need to deliver actionable analytics directly to your customers or operational teams, but your current infrastructure can't support it. You lack the solid dimensional model and scalable architecture required for a high-performance embedded analytics experience. ## A Unified Platform for Compliance, Analytics, and Growth We help you build a modern data foundation that meets the rigorous demands of the financial services industry. Our solutions are built on a secure, scalable Google Cloud architecture designed to unify your data and accelerate your time-to-insight. ### A Foundation of Trust & Compliance We architect your data platform with security and governance at its core. We help you unify data from disparate systems to create a single source of truth for all your financial, operational, and customer data, ensuring consistency and auditability. ### A 360-Degree View of the Customer We build the data models that connect the dots across the entire customer lifecycle. From initial acquisition and onboarding to transactions and support interactions, we enable you to track LTV, model churn, and fine-tune your retention strategies at scale. ### Powerful In-App & Embedded Analytics Deliver valuable insights directly to your customers. We have deep expertise in building the solid, dimensional data models required to power high-performance embedded analytics tools like ThoughtSpot, ensuring your users get fast, reliable, and actionable data. ### Data Team Enablement & Best Practices We don't just build your platform; we enable your team. We work alongside your data analysts and engineers, sharing our proven delivery workflows and best practices to give them the confidence and skills to own and extend the platform. ## Our Expertise in Action: Enabling Growth for FinTech Leaders We have a proven track record of helping FinTech companies build the data foundations they need to scale securely and effectively. ![Ageras Group](https://rittmananalytics.com/lovable-uploads/ec80f8d6-7b91-43e0-bd0e-c5cced02ced4.png) ### Accelerating Analytics Delivery for a Professional Services Platform The Ageras Group needed to mature their analytics pipeline and establish a single source of truth across their financial services brands. We partnered with their new analytics team to implement a structured dbt and Looker process, delivering high-quality data for data-driven decisions. [Read Case Study →](https://rittmananalytics.com/case-studies/ageras-group) Rebtel ### Building Real-Time Analytics for Global Communications Rebtel, a leading international calling service, needed to modernize their data infrastructure to support real-time analytics and improve customer experience. We helped them build a scalable data platform that powers their global communications network with reliable, actionable insights. [Read Case Study →](https://rittmananalytics.com/case-studies/rebtel) --- # AI & Predictive Analytics | Rittman Analytics Canonical: https://rittmananalytics.com/solutions/ai-predictive-analytics URL: https://rittmananalytics.com/solutions/ai-predictive-analytics Description: Innovate faster with AI-powered analytics. Transform data discovery with intelligent systems that understand your data and deliver insights automatically. --- Innovate faster and automate your analytical processes Transform how your organization discovers insights with AI-powered analytics. Build intelligent systems that understand your data, answer complex questions, and deliver actionable recommendations automatically. ## Innovate Faster and Automate Your Analytical Processes Boost your organization's productivity by automating insight generation and transforming static dashboards into intelligent, conversational experiences. Our AI solutions integrate cleanly with your existing data stack, delivering immediate value while ensuring data security and governance. ![Conversational Analytics Timeline](https://rittmananalytics.com/lovable-uploads/578549fe-fb0d-4073-81ca-ccfae589f5ba.png) ### AI-Powered Dashboards Transform static reports into intelligent summaries that automatically highlight key insights and trends ![Decision Maker Rating System](https://rittmananalytics.com/lovable-uploads/b7aaa5ac-9b63-44ce-a81b-aad2f2c73b52.png) ### Conversational Analytics Ask questions in natural language and get instant answers from your data warehouse ![AI Assistant Interface](https://rittmananalytics.com/lovable-uploads/f240e461-f114-4897-a40b-533fab00bc75.png) ### Automated Classification Leverage AI to automatically categorize, score, and enrich your business data ![AI Analytics Dashboard](https://rittmananalytics.com/lovable-uploads/c1733de7-f47e-40a7-a201-a5059b997010.png) ### Governed AI Implementation Deploy AI solutions with enterprise-grade security, monitoring, and compliance controls Powered by modern AI technologies and frameworks by: ![Gemini](https://rittmananalytics.com/lovable-uploads/29dc3022-a63a-468b-b8a3-d5d31ceb8a44.png) ![Perplexity](https://rittmananalytics.com/lovable-uploads/48946f92-8cd4-4613-97a6-51776a28b490.png) ![Anthropic](https://rittmananalytics.com/lovable-uploads/5ddcc8e2-a21f-483c-8ca1-3765ea035a83.png) ![LangChain](https://rittmananalytics.com/lovable-uploads/aa112ad6-327f-44f6-88b6-f41cc11321b2.png) ![OpenAI](https://rittmananalytics.com/lovable-uploads/df7b1281-2231-4f88-b67f-ae08c73d1649.png) ![Vertex AI](https://rittmananalytics.com/lovable-uploads/fc511cb0-ab0a-4495-88ff-00bb3b5ba276.png) ## Your Agents are Ready. Your Data Stack Isn't. Analytics agents, like your users, have to find the one, correct answer. Our consulting services help organisations like yours build future-proof data stacks that return the right answer, every time, to both humans and agents. ![Agentic analytics architecture diagram showing data foundation, semantic layer, skills, evals and MCP server, and consumers](https://rittmananalytics.com/lovable-uploads/agentic_data_platform.png) AI burned away the manual effort of data engineering, and what's left is architectural expertise. We use our Wire Framework to accelerate the build of agentic analytics data stacks and focus entirely on strategic value. [Explore the Wire Framework](https://rittmananalytics.com/services/wire-framework) Your Team, Multiplied by AI ## AI that earns the right to be trusted. We don't connect Claude or Gemini to your warehouse and call it transformation. We build AI on a governed semantic model, with the guardrails that make every answer one your team can defend. - Grounded in a semantic model Every metric means one thing, so AI answers match the board pack. - Built with your analysts, not around them They keep the judgment; AI takes the busywork. - Governed, auditable, defensible Know where every number came from before you act on it. ![Analyst + assistant chat interface showing a verified AI answer grounded in the governed semantic model](https://rittmananalytics.com/lovable-uploads/analyst-assistant-mockup.png) ## Before You Build, You Need a Blueprint Generative AI promises to transform industries, but implementing it successfully requires more than just access to an LLM. Without a clear strategy, a mature data foundation, and the right in-house skills, AI initiatives often lead to wasted resources, misaligned efforts, and untrustworthy results. Our comprehensive AI Readiness Assessment helps you evaluate your organization's capabilities across four key dimensions that determine AI success: ### Strategic Goals Do you have specific, value-driven use cases for AI that align with business objectives? ### Data Preparedness Is your data accessible, integrated, and governed effectively across the organization? ### Innovation Culture Does your leadership actively promote innovation and evaluate new technological initiatives? ### Team Capabilities Does your team have the necessary expertise in AI/ML and familiarity with AI ethics? --- # Marketing Analytics Solutions - Multi-Touch Attribution & ROI | Rittman Analytics Canonical: https://rittmananalytics.com/solutions/marketing-analytics URL: https://rittmananalytics.com/solutions/marketing-analytics Description: Connect marketing spend to revenue with advanced multi-touch attribution, campaign ROI analysis, and customer journey analytics. Unify your marketing data for actionable insights. --- Your Data Can't Prove It. You're running campaigns across multiple channels, but your data is trapped in silos. Spreadsheets can't connect ad spend to revenue, and a last-click-only view is hiding the true value of your marketing efforts. We build modern marketing analytics platforms that unify your entire data stack. Our solutions deliver a single source of truth for attribution, ad spend ROI, and customer behavior, empowering you to make smarter, faster, data-driven decisions. [See client results ›](https://rittmananalytics.com/case-studies) 45 minutes · no obligation · you'll speak with a founder, not sales Powering analytics for high-growth marketing and e-commerce teams like: [![Conversion Interactive](https://rittmananalytics.com/lovable-uploads/6fe9e326-335e-4023-9fa6-9869257b511d.png)](https://rittmananalytics.com/case-studies/conversion-interactive)[![Rixo](https://rittmananalytics.com/lovable-uploads/d31fb621-3b64-42a8-b80a-dcd5b349fa1c.png)](https://rittmananalytics.com/case-studies/rixo)[![Insighta](https://rittmananalytics.com/lovable-uploads/dc3bd3b8-cc99-41d1-ab62-6f8502012b06.png)](https://rittmananalytics.com/case-studies/insighta) ## Why Your Marketing Data is Working Against You If you're a marketing leader, this probably sounds familiar. Your team is delivering results, but your reporting is a mess of disconnected data and manual work that fails to tell the full story. ### You Can't Connect Spend to Revenue Your Google Ads, Facebook Ads, and other platform data lives in one place, while your sales and revenue data from Shopify or your CRM lives somewhere else. Manually stitching this together in spreadsheets is slow, error-prone, and unsustainable. ### You're Blind to the Full Customer Journey Relying on platform-native analytics forces you into a last-click view of the world. You have no way to see how different channels, from content marketing to paid social, work together to influence a conversion over time. ### You Lack a Single, Trusted View of the Customer Data from your event tracking tools (like Segment or Snowplow), your product database, and your support tools are not connected. You can't see how user behavior actually impacts lifetime value or repeat purchases. ## A Unified View of Your Marketing Performance Instead of building from scratch, leverage our production-ready solutions to centralize your marketing data and unlock powerful insights. Our frameworks are built on Google Cloud and Looker technology and designed to integrate cleanly with marketing technology data sources and applications. ![Multi-Touch Marketing Attribution](https://rittmananalytics.com/lovable-uploads/3d595151-24c9-4fb2-aafb-3310497d854f.png) ### Multi-Touch Marketing Attribution Go beyond last-click. Our dbt package deploys multiple industry-standard attribution models (first-touch, time-decay, linear) to give you a complete picture of how each channel contributes to revenue, customer LTV, and new user acquisition. ![Ad Spend & Campaign ROI](https://rittmananalytics.com/lovable-uploads/f925f00f-c12c-400d-b75c-7fe278f7b46a.png) ### Ad Spend & Campaign ROI Finally get a true measure of your return on investment. We centralize ad spend and performance data from Google Ads, Facebook Ads, and more, connecting it directly to your conversion and revenue data to deliver clear, actionable ROI analysis. ![Composable Customer Data Platform (CDP)](https://rittmananalytics.com/lovable-uploads/3de0759b-637d-4828-b31a-46fa6cb38725.png) ### Composable Customer Data Platform (CDP) Create a single source of truth for every customer. We unify data from your event streams, CRM, and product databases to build a comprehensive 360-degree view, enabling deeper segmentation and personalization. ![Customer Journey Analytics & Orchestration](https://rittmananalytics.com/lovable-uploads/87aea2e7-f63d-497e-ae9f-dfc9ae1ad467.png) ### Customer Journey Analytics & Orchestration Understand the complete customer path to conversion with advanced identity resolution and journey mapping. Our solutions track every touchpoint across channels, resolve customer identities, and enable real-time journey orchestration for personalized experiences. ## Our Expertise in Action: Powering Analytics for Conversion Interactive Conversion Interactive, a full-service ad agency for the transportation industry, needed a world-class analytics infrastructure to track the entire driver lead funnel, from initial ad click to final hire. > "Rittman Analytics' DEEP TECHNICAL expertise with the modern data stack increased our data team's capabilities." Kelley Walkup, President and CEO, Conversion Interactive Agency ### Our Partnership Delivered: A Unified Data Stack built on dbt, Google BigQuery, and Rudderstack to centralize data from disparate ad platforms and recruitment tools. End-to-End Funnel Visibility, allowing them to drill down into metrics for each stage of the driver lead funnel, Awareness, Consideration, and Conversion. Actionable Recruiter & Fleet Insights, providing a clear understanding of what activities best support recruiting and retention efforts in real-time. [Read the Full Conversion Interactive Case Study](https://rittmananalytics.com/case-studies/conversion-interactive) --- # SaaS & Mobile App Analytics Solutions | Rittman Analytics Canonical: https://rittmananalytics.com/solutions/saas-mobile-analytics URL: https://rittmananalytics.com/solutions/saas-mobile-analytics Description: Transform your SaaS metrics with advanced mobile app analytics. Connect user behavior to revenue impact and optimize the full customer lifecycle. --- Build a Product That Wins on Insights. Your app is generating millions of events, but are you measuring what truly matters? Without a unified view of user behavior and commercial data, your roadmap is driven by anecdote, not evidence. We partner with innovative SaaS and mobile app companies to transform product usage data into your biggest competitive advantage. We build the data foundations that connect feature adoption to revenue, empowering your product, marketing, and finance teams to make smarter decisions, faster. The data partner for innovative product teams at: [![Ageras Group](https://rittmananalytics.com/lovable-uploads/26586569-3def-44fd-b25d-f3b926dd50e7.png)](https://rittmananalytics.com/case-studies/ageras-group)[![Popsa](https://rittmananalytics.com/lovable-uploads/0efdef7b-4690-4f31-8c43-0fa67b294f2f.png)](https://rittmananalytics.com/case-studies/popsa)[![ROUVY](https://rittmananalytics.com/lovable-uploads/bf807715-48cb-49e0-8424-9e6174a03664.png)](https://rittmananalytics.com/case-studies/rouvy)[![Insighta](https://rittmananalytics.com/lovable-uploads/dc3bd3b8-cc99-41d1-ab62-6f8502012b06.png)](https://rittmananalytics.com/case-studies/insighta) ## Why Most Product Analytics Fails to Deliver Value Many fast-growing SaaS companies invest in product analytics tools but struggle to connect the data to meaningful business outcomes. This leads to a common set of challenges that keeps your team guessing. ### You're Drowning in Data, But Starving for Insights You have detailed event tracking, but the data is siloed from your subscription, CRM, and financial systems. You can't see what users are doing, but you can't see why or what the commercial impact is. ### Your Team is Building on a Brittle Foundation Your data stack is a patchwork of legacy systems and custom connectors. It can't handle large datasets, leading to slow processing times, inconsistent metrics, and a lack of trust in the numbers from your product and leadership teams. ### You Can't See the Full Customer Lifecycle You struggle to connect user acquisition channels to long-term engagement and LTV. Your team can't reliably answer critical questions like, "Which marketing channels bring in our most valuable users?" or "Which features drive the highest retention?" ## A Unified Platform for Product, Marketing, and Finance We help you build a single source of truth that aligns your entire organization around a shared understanding of your customers and your business. Our solutions are built on Google Cloud technology and designed to deliver clarity and sustainable growth. ### Deep Product & User Analytics Go beyond pageviews. We help you model and track feature adoption, user flows, and behavioral cohorts to understand where customers find value, where they experience friction, and which segments are primed for growth. ### End-to-End Commercial Insights We build the data pipelines that connect the dots. By unifying product usage data with financial data from sources like Chargebee, we provide a clear, end-to-end view of customer acquisition cost (CAC), LTV, and channel effectiveness. ### Powerful Embedded Analytics Deliver actionable insights directly to your customers. We have deep expertise in building the solid, dimensional data models and scalable infrastructure required to power high-performance embedded analytics within your application. ### Modern, Scalable Data Infrastructure We architect and build the solid data foundation you need to scale. From implementing first-party event tracking with Snowplow to building a dimensional data warehouse with dbt and Looker, we ensure your infrastructure is secure, reliable, and ready for the future. ## Our Expertise in Action: Powering Growth for SaaS Leaders We have a proven track record of helping SaaS and mobile app companies build the data foundations they need to innovate and scale with confidence. ![Client Logo](https://rittmananalytics.com/lovable-uploads/0efdef7b-4690-4f31-8c43-0fa67b294f2f.png) ### Delivering an End-to-End View of a Mobile Commerce Business Popsa needed to move beyond bottlenecks in their reporting. We implemented Looker and built out a financial data lake, extending it to include marketing spend to give their team a complete view of customer acquisition cost, revenue, and channel effectiveness. [Read Case Study →](https://rittmananalytics.com/case-studies/popsa) ![Client Logo](https://rittmananalytics.com/lovable-uploads/bf807715-48cb-49e0-8424-9e6174a03664.png) ### Rebuilding a High-Performance Data Engine for a Global App ROUVY's legacy stack couldn't keep up with their growth. We partnered with them to design and build an entirely new, best-in-class data platform using Snowplow, dbt, and Looker, empowering their team to own their data future. [Read Case Study →](https://rittmananalytics.com/case-studies/rouvy) --- # Cube Partner - Semantic Layer Experts - Rittman Analytics Canonical: https://rittmananalytics.com/partners/cube URL: https://rittmananalytics.com/partners/cube Description: Rittman Analytics is an official Cube partner, specializing in semantic layer implementations, data modeling, and modern data stack architecture with Cube.dev technology. --- Experts in the Semantic Layer ![Cube Partner Network](https://rittmananalytics.com/lovable-uploads/82155628-cf07-432b-9221-8ae6dcaa2d35.png) Rittman Analytics is a boutique analytics consultancy specializing in the modern data stack. As a founding member of the Cube Partner Network, our team provides expert consulting services to help organizations get started with Cube, either as part of a wider data strategy engagement or as a focused project to implement Cube's semantic layer on top of an existing data warehouse. ## What Is Cube? Trust your insights. Consume data from any source, organize it into consistent metrics, and use it with every data app. ### Universal Semantic Layer + Agentic Analytics **Best-of-Breed Semantic Layer with No Platform Lock-In:**A semantic layer is a middleware between your data source and your data application. Cube is an API-first, four-part semantic layer that enables data engineers and developers to make their data consistent, secure, performant, and accessible across every application. **Works with All SQL-Enabled Data Sources:**Designed to work with all SQL-enabled data sources including Snowflake, Google BigQuery, Presto and Postgres, Cube has a built-in caching engine to provide sub-second latency and high concurrency for API requests. **Built for embedding and data apps:**Headless, API-first, and decoupled from visualizations, You can use Cube with any charting library to build your own custom UI, or connect existing dashboarding and reporting tools to Cube and expose your data to every application via REST, GraphQL, and SQL APIs. **D3 Agentic Analytics Platform:**Composed of AI-powered agents that specialize in data engineering and data analysis, these new capabilities capitalize on a growing market demand for trustworthy, autonomous AI systems that are explainable, governed, and enterprise-ready. ### Proof in Practice ![RIXO Store Front](https://rittmananalytics.com/lovable-uploads/b6da3bf6-740b-43ee-baa7-8bd1754d549f.png) ![Osprey Charging Network](https://rittmananalytics.com/lovable-uploads/29e54000-f903-4152-aada-f143b45338da.png) #### How Rittman Analytics Helped Osprey Scale Their Data Platform Osprey Charging partnered with Rittman Analytics to scale its data platform during rapid expansion, installing 800 chargers in nine months. We delivered a modern analytics stack using Snowflake, dbt, Fivetran, Preset, and Cube. [Read the Full Case Study](https://rittmananalytics.com/case-studies/osprey) ## Cube Quickstart Package Would you like to run Cube on-top of your Snowflake, Google BigQuery or other cloud data warehouse but don't have the necessary data engineering resources or know-how to develop a production-quality Cube semantic model? We can help. Our data and analytics engineering team can setup Cube for you, build your semantic model and integrate it with your BI tool or help embed it in your SaaS application. We provide your engineers and data team with the training they need to get up to speed, or manage and support Cube for you. ![Rittman Analytics Cube Quickstart Timeline](https://rittmananalytics.com/lovable-uploads/190e3b0f-ee1b-4969-af81-e6f891e76be6.png) ### What Is Included Cube semantic model design, implementation blueprint and documentation Deployment of your Cube semantic model into the cloud of your choice using Cube Cloud Configuration of access control, row- and column-level security Connection of Cube to your SQL-based BI tool (e.g., Tableau, Superset, etc), notebook (e.g., Deepnote, Hex, etc.) or collaboration with your engineering team to integrate Cube into your application via REST API Query optimization using Cube's in-memory cache, queue management, and configurable pre-aggregations Documentation and enablement for your data team or engineering team ### What Do We Need? To deliver our Quickstart package within four weeks, you will need to have already: Implemented a cloud data warehouse, structured appropriately for ad-hoc analytics queries Chosen and implemented your SQL-based BI tool, or developed an application into which Cube will be embedded Identified a use-case suitable for use with Cube and capable of being delivered within four weeks Not ready yet to deploy Cube? Don't worry - we can help get you there, and then work with you to deliver the Cube Quickstart! ## Ready to Build a High-Performance Semantic Layer? Let's discuss how our expert implementation of Cube can accelerate your roadmap and deliver trusted, consistent metrics to your users. Schedule a free, no-obligation call to scope your project. --- # Dagster Partner - Modern Data Orchestration - Rittman Analytics Canonical: https://rittmananalytics.com/partners/dagster URL: https://rittmananalytics.com/partners/dagster Description: Rittman Analytics partners with Dagster to deliver modern data orchestration solutions. Expert implementation of Dagster pipelines, data ops, and workflow automation. --- Modern Data Orchestration ![Dagster Implementation Partner](https://rittmananalytics.com/lovable-uploads/5d3084da-f791-4f73-9738-ce1f8a9651a0.png) As one of the founding members of the Dagster Partner Network, we recommend Dagster as the cloud-native orchestrator for the whole development lifecycle with integrated lineage and observability, a declarative programming model, and best-in-class testability. --- # dbt Labs Partner - Analytics Engineering Excellence | Rittman Analytics Canonical: https://rittmananalytics.com/partners/dbt-labs URL: https://rittmananalytics.com/partners/dbt-labs Description: Transform your data and accelerate your insights with Rittman Analytics, a registered dbt Labs partner. Expert dbt implementation, custom packages, and analytics engineering services. --- Transform your data. Accelerate your insights. ![dbt Labs Registered Partner](https://rittmananalytics.com/lovable-uploads/36ee70a4-8ae8-465a-aa19-438e8c12a779.png) dbt (Data Build Tool) is a transformation workflow that helps data teams get more work done while producing higher quality results. You can use dbt to modularize and centralize your analytics code, while also providing your data team with guardrails typically found in software engineering workflows. ## Specialists in Data Centralisation using dbt As a dbt Labs partner since 2019, Rittman Analytics brings deep expertise and proven methodologies to every dbt implementation. ### Our Expertise & Credentials #### 7+ years of dbt experience Adopting dbt for all our major client projects from when dbt Labs was called Fishtown Analytics #### Client success stories Proven implementations across fintech, legal services, and finance sectors #### Cross-platform expertise Proven implementations across Google BigQuery, Snowflake, and Oracle Autonomous Data Warehouse #### Advanced integration patterns Sophisticated entity resolution and deduplication across multiple data sources #### Open-source leadership 352+ GitHub stars across our dbt framework contributions, including the RA Data Warehouse framework used by teams worldwide #### Modern data stack integration Seamless connection with Fivetran, Stitch, Segment, Looker, and other leading data tools ### Proof in Practice ![Data analytics team working](https://rittmananalytics.com/lovable-uploads/759b77cf-7e28-4aed-bab0-0af7986103a4.png) ![Torticity Logo](https://rittmananalytics.com/lovable-uploads/1dd119e2-ad0b-4af4-9f1d-b02be5268aec.png) #### Torticity Uses Rittman Analytics To Create a Foundation for In-App Analytic Insights Rittman Analytics were engaged to increase the capabilities of Torticity's data team, understanding their business process and model and transform their data in Snowflake Data Warehouse using dbt and Rittman Analytics' delivery workflow. [Read the Full Case Study](https://rittmananalytics.com/case-studies/torticity) ## Consulting Packages and Tools for dbt Accelerate your delivery and time-to-value with our dbt packages and open-source tools ![RA Warehouse for dbt](https://rittmananalytics.com/lovable-uploads/67e6c9fe-fdc1-44a3-aef7-d6216fac3e14.png) ### RA Warehouse for dbt This dbt package contains a set of pre-built, pre-integrated Load and Transform dbt models for common SaaS applications [View on GitHub](https://github.com/rittmananalytics/ra_data_warehouse) ![RA Attribution for dbt](https://rittmananalytics.com/lovable-uploads/36f5d506-4b7e-4679-bd96-f88be2a6ae92.png) ### RA Attribution for dbt A multi-touch, multi-cycle marketing attribution model built using dbt and with dashboard content for Looker and Lightdash [View on GitHub](https://github.com/rittmananalytics/ra_attribution) ![Droughty](https://rittmananalytics.com/lovable-uploads/a66bebcd-d2ee-4366-a574-af63a3a70cf1.png) ### Droughty droughty is an analytics engineering toolkit that automates the creation of dbt models, tests, QA processes and LookML content [View on PyPI](https://pypi.org/project/droughty/) ![Data Platform Implementation](https://rittmananalytics.com/lovable-uploads/6e7296ba-e9aa-483d-9e13-c090b4a69557.png) ### Data Platform Implementation Introduce a modern analytics workflow and software engineering best practices with our services packages based around dbt Core [Learn More](https://e7026abc-5584-4623-aa1c-417b22103092.lovableproject.com/services/data-platform-development) ## Ready to Transform Your Data with dbt? Let's discuss how our proven dbt expertise can accelerate your data transformation journey and deliver trusted, consistent analytics to your organization. Schedule a free, no-obligation call to explore your project. --- # Embeddable Partnership - Advanced Analytics Solutions | Rittman Analytics Canonical: https://rittmananalytics.com/partners/embeddable URL: https://rittmananalytics.com/partners/embeddable Description: Partner with Rittman Analytics and Embeddable to deliver advanced embedded analytics solutions. Expert implementation and support services. --- Enabling Remarkable Analytics Experiences ![Embeddable](https://rittmananalytics.com/lovable-uploads/403670aa-4f0d-4615-8311-edcbdca80901.png) Embeddable is a developer toolkit for building fast, interactive, fully-custom analytics experiences directly into your app. As a consulting partner to Embeddable, we specialize in preparing and structuring your data to make it analytics-ready for Embeddable dashboards. ## What Is Embeddable? The Toolkit for Building World-Class Embedded Analytics **Build Beautiful, Native Analytics Experiences:**Embeddable is a developer-first toolkit designed to help you create fast, interactive, and fully custom analytics experiences that integrate cleanly into your application. It gives you the power to build pixel-perfect dashboards that look, feel, and behave exactly like the rest of your platform. **Powered by Cube's Semantic Layer:**Embeddable leverages the power of Cube, an open-source semantic layer that sits between your data warehouse and your application. Cube organizes your raw data into consistent, reusable business definitions and metrics, providing a single source of truth for all your analytics. **Achieve Lightning-Fast Performance:**By using Cube's powerful caching and pre-aggregation engine, Embeddable can deliver insights with lightning-fast loading speeds. This ensures your customers get an exceptional user experience, even when querying large datasets, while keeping your database costs low. **Separate Your App's Logic from Your Analytics:**Cube acts as a decoupling layer, allowing your analytics to evolve independently from your application's production database. You can stage your data in a separate data warehouse and use tools like dbt for transformation, ensuring that analytical queries never impact your app's performance. **Deliver a Consistent, Governed Data Experience:**The Cube semantic layer ensures that every metric and dimension is defined once and reused everywhere. This guarantees data consistency across all of your embedded analytics, providing your customers with reliable, trustworthy insights they can act on with confidence. ### Proof in Practice ![Smart Industrial Control System Dashboard](https://rittmananalytics.com/lovable-uploads/8501a8fd-4d8e-4009-bb8b-93a9e3bf3cef.png) ![Facility Solutions Group](https://rittmananalytics.com/lovable-uploads/edd34324-8c92-46e6-b8d3-c19404e2bdef.png) #### How Rittman Analytics Delivered Embedded Analytics for FSG Rittman Analytics rapidly accelerated Facility Solutions Group's (FSG) data analytics initiative, transforming a delayed and under-resourced project into a structured and effective embedded analytics solution within weeks. By implementing the Embeddable platform, building a solid data pipeline, and collaborating closely with FSG's teams, Rittman Analytics enabled scalable KPI tracking and intuitive client-facing dashboards. [Read the Full Case Study](https://rittmananalytics.com/case-studies/fsg-smart-buildings) ## Embeddable Quickstart Packages Rittman Analytics partners with Embeddable and Cube to help your team get your data into shape, modelled and transformed and ready for use as the datasets for your Embeddable data visualizations. ![Embeddable Jump Start](https://rittmananalytics.com/lovable-uploads/18cf3ef4-2e8a-4b1c-b9c5-cbcfc078d85e.png) ### Embeddable Jump Start £4,950.00 ![Embeddable Jumpstart Plus](https://rittmananalytics.com/lovable-uploads/64d4de0b-7a09-4c80-b7d0-27a96edee6d0.png) ### Embeddable Jumpstart Plus £9,950.00 ![Embeddable Custom Development Sprint](https://rittmananalytics.com/lovable-uploads/55136fe1-a4d7-467d-88d7-1aaf32d301e2.png) ### Embeddable Custom Development Sprint from £7,950.00 To find-out more about our Embeddable Quickstart Package or other consulting services for the modern data stack, either contact us at info@rittmananalytics.com, download our services brochure or press the button below to arrange a free, no-obligation initial Discovery call. ## Ready to Build World-Class Embedded Analytics? Let's discuss how our expert implementation of Embeddable can accelerate your roadmap and deliver trusted, consistent data visualisations to your users. Schedule a free, no-obligation call to scope your project. --- # Fivetran Partnership - Data Integration Solutions | Rittman Analytics Canonical: https://rittmananalytics.com/partners/fivetran URL: https://rittmananalytics.com/partners/fivetran Description: Partner with Rittman Analytics and Fivetran for seamless data integration with 700+ connectors. Expert ELT solutions and data pipeline consulting. --- Integrate and move data with 700+ data connectors. ![Fivetran Systems Integrator Partner](https://rittmananalytics.com/lovable-uploads/8cdef00a-4cc5-4513-aef5-fe76e77bbbf1.png) Automatically, reliably and securely move data from 700+ sources including SaaS applications, databases, ERPs, and files to data warehouses, data lakes, and more. --- # Google Cloud & Looker Partnership - Advanced Analytics Solutions | Rittman Analytics Canonical: https://rittmananalytics.com/partners/googlecloud URL: https://rittmananalytics.com/partners/googlecloud Description: Expert Google Cloud and Looker partnership for advanced analytics. Generative AI integration, BigQuery optimization, and comprehensive platform implementation. --- Experts in Looker Implementation ![Google Cloud Partner with Looker, BigQuery, and Vertex.ai logos](https://rittmananalytics.com/lovable-uploads/692d8358-47b0-4d52-b432-c5901df0cde9.png) As a Google Cloud Specialized Data Analytics Partner, we provide expert consulting, implementation, and support for the entire Google Cloud data ecosystem. From building a powerful data warehouse on BigQuery to enabling true self-service analytics with Looker, our deep expertise ensures you unlock the full value of your investment. ## Certified Experts with a Proven Track Record We have been delivering world-class analytics solutions on Google Cloud and Looker since 2016. Our team of certified Looker Developers and Google Cloud experts has a long history of successful implementations for clients across the UK, Europe, and North America. ### Our Expertise & Credentials **Google Cloud Specialized Partner:**Officially recognized by Google for our deep technical expertise and proven customer success in Data & Analytics. **Seven Years of Looker Implementation:**Extensive, hands-on experience delivering complex projects for both internal BI and embedded "Powered by Looker" use cases. **Full Ecosystem Mastery:**We are experts in the entire Google Cloud data stack, including BigQuery, Cloud Storage, Dataflow, Cloud Composer, and Vertex AI. **AI-Powered Analytics Pioneers:**Early adopters of AI for Looker, building Conversational Analytics, Vertex AI-powered dashboards, and our open-source Langchain Looker Agent for natural language querying. **Community Leaders:**Organizers of the [London Looker Meetup](https://www.meetup.com/london-looker-meetup/), bringing together the UK's Looker community to share knowledge and best practices. **Trusted by Leading Brands:**We have a long track record of helping high-growth companies in SaaS, E-commerce, and FinTech build and scale their analytics on Google Cloud. [View Our Google Cloud Partner Profile](https://rittmananalytics.com/#) ### Proof in Practice ![RIXO Store Front](https://rittmananalytics.com/lovable-uploads/0e44d9a1-b3b2-4313-85d1-abfa7c6538cc.png) ![RIXO Logo](https://rittmananalytics.com/lovable-uploads/16a19fa4-6cbf-490d-b0b9-e70afaef3a36.png) #### How Rittman Analytics Helped RIXO Create a Single View of Trading Operations We partnered with the global fashion brand to centralize their data from NetSuite, Shopify, and multiple ad platforms into Google BigQuery, building a set of automated Looker dashboards to power their critical weekly trade reporting. [Read the Full Case Study](https://rittmananalytics.com/case-studies/rixo) ## Services Tailored for Google Cloud & Looker Whether you're just getting started or looking to optimize an existing implementation, we offer a range of services and fixed-price packages to help you succeed. ![Generative AI & Looker Integration](https://rittmananalytics.com/lovable-uploads/59292a1e-36ff-48f7-b751-6798533aebfc.png) ### Generative AI & Looker Integration Supercharge your analytics with AI. We build custom solutions using Vertex AI and LangChain to automate dashboard summaries, classify your data, and enable users to query Looker with natural language. ![Expert Consulting & Team Enablement](https://rittmananalytics.com/lovable-uploads/56de72b4-1769-4094-9731-d1d0361b5461.png) ### Expert Consulting & Team Enablement Leverage our senior architects and developers to solve your most complex challenges. We provide expert consulting on platform architecture, LookML modeling, and performance tuning, along with hands-on training for your team. ![Full Platform Implementation](https://rittmananalytics.com/lovable-uploads/76aec625-c258-4068-86a8-f8344146ef6c.png) ### Full Platform Implementation A comprehensive service to build your entire data platform on Google Cloud. We handle everything from the initial data strategy and BigQuery warehouse setup to Looker development and team enablement. ![Looker Quickstart Packages](https://rittmananalytics.com/lovable-uploads/57727423-bc86-418f-abff-5875818db558.png) ### Looker Quickstart Packages Fixed-price, defined scope engagements designed to help you quickly evaluate and implement Looker. Includes our Looker Jumpstart, Proof of Concept, and Generative AI packages to accelerate your time-to-value. ## Ready to Maximize Your Google Cloud Investment? Let's discuss how our specialized expertise can help you build a world-class data platform, solve your most complex challenges, and empower your team with reliable, self-service analytics. --- # RudderStack Partnership - Customer Data Platform Solutions | Rittman Analytics Canonical: https://rittmananalytics.com/partners/rudderstack URL: https://rittmananalytics.com/partners/rudderstack Description: Partner with Rittman Analytics and RudderStack for comprehensive customer data platform implementation and data infrastructure solutions. --- Collect & Activate your Customer Data ![RudderStack Solutions Partner](https://rittmananalytics.com/lovable-uploads/6ce502c7-3e17-4a0f-af7c-e0b043a2047c.png) RudderStack makes it easy to collect and send customer data to the tools and teams that need it, streaming clean first-party customer event data into your data warehouse, data lake and SaaS services. ## What is RudderStack? **Collect and send customer data to the tools and teams that need it:**RudderStack streams clean first-party customer event data into your data warehouse, data lake and SaaS services. **Cost-effective alternative to Segment:**Based around an open-source re-implementation of Segment's Javascript and other SDKs, RudderStack can capture customer interactions across all your mobile channels and route it to all your marketing and customer SaaS application destinations. **Warehouse Native CDP:**RudderStack is built to help data teams deliver value across the entire data activation lifecycle, from collection to unification and activation. **Event Stream and Cloud Extract ETL pipelines:**RudderStack makes it easy to collect data from every website, app and cloud tool that stores customer data. Standardized schemas, data governance tools and powerful transformations ensure that data is consistent, clean and trustworthy when it's delivered. **Profiles feature for complete customer view:**RudderStack's Profiles feature leverages standardized schemas to significantly reduce the time it takes to build a complete view of the customer, including an identity graph and rich customer features. **Industry-Standard Tracking Interface:**Rittman Analytics are also Implementation Partners for Segment and Hightouch, all of which have compatible tracking libraries. We can implement any of these tracking services for you, or migrate you from one to the other quickly and with no downtime or resetting of customer tracking IDs. ![Customer Data Platform Architecture](https://rittmananalytics.com/lovable-uploads/a6e81b55-af1d-40ec-ba8e-773d286e712a.png) ![RudderStack Connections Interface](https://rittmananalytics.com/lovable-uploads/20fa321c-4586-44cf-91ab-539f2685a8ed.png) ## RudderStack Quickstart Package Would you like to get started with RudderStack but don't have the necessary data engineering resources or know-how to configure a production-quality RudderStack instance and start loading event data into your customer data warehouse? ### We can help. With our RudderStack QuickStart Package, you receive a fixed number of consulting hours from Rittman Analytics at a pre-set cost. These hours are then applied to key deliverables to help you get the highest possible ROI out of your customer data strategy. #### We will work with you on deliverables including: - Tracking plan creation / migration - Warehouse schema review and modeling - Sources and destinations prioritization - Workflow architecture and design ### 4-Week Implementation Timeline ##### Week 1 Project kickoff, strategy, discovery, and KPI review ##### Week 2 Create Tracking Plan and review data strategy ##### Week 3 Implementation with Client's Technical Team and RudderStack SDK setup ##### Week 4 Implementation Review, QA, and project handoff to Customer Success team ## Ready to Transform Your Customer Data Strategy? Let's discuss how RudderStack can streamline your customer data collection and activation. --- # SnowCatCloud Partner - Customer Analytics - Rittman Analytics Canonical: https://rittmananalytics.com/partners/snowcatcloud URL: https://rittmananalytics.com/partners/snowcatcloud Description: Rittman Analytics partners with SnowCatCloud to deliver advanced customer analytics solutions. Expert implementation of customer data platforms and behavioral analytics. --- Open Source Customer Data Infrastructure ![SnowCatCloud Consulting Partner](https://rittmananalytics.com/lovable-uploads/d6fba808-d012-477a-98db-5a52018a2f28.png) SnowcatCloud provides Snowplow-compatible open-source infrastructure to collect, enrich, load and integrate behavioural data at scale. As one of only two official SnowcatCloud Technology and Consulting Partners, Rittman Analytics accelerates your implementation of SnowcatCloud and a Modern Data Stack. ## Consulting Services for Customer Analytics Accelerate your Implementation of SnowCatCloud, Segment, Rudderstack or Hightouch Events ![Customer Data Platform](https://rittmananalytics.com/lovable-uploads/a9e8c09f-2c17-4243-be80-fc7bfbb6aeec.png) ### Warehouse-Centric Customer Data Platforms Build comprehensive customer data platforms using SnowCatCloud to collect, unify, and activate your customer data across all touchpoints and channels. ![Identity Resolution & Customer Journey](https://rittmananalytics.com/lovable-uploads/4c574023-f11e-4f01-a0fa-c1facbfd9deb.png) ### Identity Resolution & Customer Journey Implement advanced identity stitching and customer journey mapping to create a complete view of your customers across devices and interactions. ![Digital Analytics & Marketing RoI](https://rittmananalytics.com/lovable-uploads/e34ed090-d08d-4d34-8c85-399b111f7589.png) ### Digital Analytics & Marketing RoI Measure and optimize your marketing performance with detailed analytics, conversion tracking, and ROI analysis powered by behavioral data. ![Marketing Attribution Models](https://rittmananalytics.com/lovable-uploads/8ab0db01-0851-47cd-92bc-c70352663ca1.png) ### Marketing Attribution Models Develop sophisticated attribution models to understand the true impact of your marketing channels and optimize budget allocation. --- # Case Study: The Ageras Group - Accelerating Analytics Delivery | Rittman Analytics Canonical: https://rittmananalytics.com/case-studies/ageras-group URL: https://rittmananalytics.com/case-studies/ageras-group Description: Learn how The Ageras Group partnered with Rittman Analytics to build trusted analytics foundations with dbt and Looker for better data-driven decisions. --- ![Ageras Group logo](https://rittmananalytics.com/lovable-uploads/ec80f8d6-7b91-43e0-bd0e-c5cced02ced4.png) # The Ageras Group Uses Rittman Analytics To Accelerate Their Analytics Delivery ![Jesper Holm Olsen](https://rittmananalytics.com/lovable-uploads/e3be4e4d-098a-4df0-b810-d93961b99eec.png) #### Jesper Holm Olsen Product Manager, Ageras Group > "Rittman Analytics helped us build out analytics and deliver high-quality data, so we have the best foundation for our data-driven decisions." ![Ageras Group team](https://rittmananalytics.com/lovable-uploads/3c7be7e5-4488-4912-b268-2e49fc654a19.png) ### Vital Stats #### Industry Professional Services #### Founded 2012 #### Services Finance and accounting, tax compliance, tax agents, legal services #### Headquarters Copenhagen, Denmark ### Rittman Analytics Services • Data strategy and architecture • Platform development • Data science & ML • Platform transformation ### Technologies Used • Looker • dbt • Amazon Redshift ## The Download The Ageras Group is the home of several brands that provide different financial offerings to help micro-businesses succeed. Billy, Meneto and Tellow provides accounting platforms and services and Ageras is a marketplace for connecting accountants with clients. ## Seeking Data Visibility The Group found itself in an all too common situation of having multiple different dashboards, created by many different people and utilising data from many different sources but without alignment on data definitions. This meant data was not trusted between stakeholders. Additionally, there was a lot of data that had not been visualised at all, resulting in the Group missing out on making data-driven decisions. In a quest to avoid the trap of comparing apples and oranges when different people create dashboards and analysis using their own definitions of the data, The Group decided to strictly follow the dbt approach for transforming data in a structured and documented way. ## Maturing The Pipeline For Data Insights With a brand new analytics team in place without a lot of experience with dbt and Looker, The Ageras Group partnered with Rittman Analytics to help get a head start on a mature process. --- # Case Study: BARK - Scaling Data Platform and Culture | Rittman Analytics Canonical: https://rittmananalytics.com/case-studies/barkbox URL: https://rittmananalytics.com/case-studies/barkbox Description: Learn how BARK scaled its data platform and culture with Rittman Analytics, migrating to BigQuery and Looker to establish a single source of truth. --- ![BarkBox logo](https://rittmananalytics.com/lovable-uploads/barkbox-logo.jpeg) # How BARK Scaled Its Data Platform and Culture with Rittman Analytics ![Michael Bishop](https://rittmananalytics.com/lovable-uploads/michael-bishop-barkbox.jpeg) #### Michael Bishop VP, Data, BARK > "Working with Rittman Analytics allowed us to deliver the intuitive, reliable data experience our teams have needed. Their expertise helped us accelerate adoption across the business with minimal friction, one of the smoothest consulting engagements of my career." ![BarkBox subscription box with dog toys and treats](https://rittmananalytics.com/lovable-uploads/barkbox-hero.png) ### Vital Stats #### Website https://www.bark.co/ #### Industry eCommerce #### Company Size 501-1,000 employees #### Headquarters New York, USA ### Rittman Analytics Services • Data Centralisation • Data Team Enablement • Data Team Modernisation • dbt Consulting & Support ### Technologies Used • Google BigQuery • Looker • dbt • Hevo ## The Download BarkBox is BARK's flagship monthly subscription, delivering themed boxes of toys and treats tailored to dogs' unique playstyles. Every month features an original collection that surprises and delights both dogs and their humans. The company's success is built on deep customer insight, playful storytelling, and a commitment to a dog-first design. With a strong omnichannel strategy and data-driven approach to personalization, BARK has created a powerful emotional connection with its customers. This foundation, paired with exceptional customer service, maximizes lifetime value, achieves high retention and maintains its position as a leader in the pet industry. ## The Challenge As BarkBox scaled, its data capabilities struggled to keep up. The company found itself in a familiar situation of a rapidly growing business built on data infrastructure designed for a startup. What had once worked for a startup environment was now a limiting factor for a fast-growing, data-rich business. - **Fragmented and Inconsistent Data:** Information was scattered across a legacy e-commerce platform, third-party systems and countless spreadsheets. Without a single source of truth, teams faced conflicting business definitions and unreliable reports. - **Manual and Inefficient Processes:** The marketing team relied heavily on manual workflows to consolidate campaign data, track performance and report on customer engagement. Pulling insights from multiple sources including ad platforms, email systems and internal databases required hours of spreadsheet work each week not only slowed down decision-making but also made it difficult to react quickly to trends, optimize campaigns in real time and accurately measure ROI. - **Limited Access to Strategic Insights:** The legacy data warehouse and complex data model made it difficult for teams to explore or analyze data independently. This lack of scalability and governance hindered self-service analytics and prevented BARK from unlocking the full potential of AI-driven insights. BARK's legacy cloud analytics infrastructure was increasingly unable to keep pace with the company's growth and evolving data needs. Query performance had degraded as data volumes scaled, creating bottlenecks for analysts and slowing time-to-insight. The platform's cost structure made scaling prohibitive, while the absence of a true semantic layer meant metric definitions varied across teams, undermining confidence in reporting and making a "single source of truth" elusive. ## The Solution Rittman Analytics took a deeply collaborative, embedded approach to help BARK build a modern, scalable data platform using our proven **Data Team Enablement Framework**. 1\. Confront the Hard Truth 2\. Create the Change 3\. Manage the Change ### Phase 1: Confront the Hard Truths Through Discovery We began with an in-depth discovery process, working closely with stakeholders across the business to fully understand the challenges of scaling BARK's data function. Through a series of collaborative workshops and technical deep-dives, we were able to uncover both strategic and operational barriers to growth. This process included: - **Comprehensive stakeholder mapping** across every business unit to understand data needs, dependencies and priorities. - **A full technical assessment** of the existing infrastructure, conducted with complete transparency and access. - **Joint problem-solving** sessions with internal teams to align on challenges and co-design potential solutions. - **A detailed capability gap analysis** identifying where people, processes and technology needed to evolve to support a modern data organization. - **A comprehensive audit** of BARK's existing legacy environment, documenting technical debt, performance constraints, and governance gaps. Following a structured multi-vendor evaluation, BigQuery and Looker were selected as the target platform, a decision aligned with BARK's broader strategic commitment to the Google Cloud ecosystem. ### Phase 2: Create the Change The team executed a full platform migration to Google BigQuery and Looker, systematically transitioning production workloads while maintaining business continuity. Over a twelve-month engagement, we delivered: - **Data Warehouse Migration:** All historical data and active pipelines were migrated to BigQuery, with Hevo and custom integrations replacing legacy ingestion processes. dbt was implemented to power multi-layered transformations, creating a modern, scalable foundation for analytics. - **Semantic Layer Implementation:** Looker replaced the legacy business intelligence platform, establishing a unified semantic layer with governed metric definitions. This provided the "single source of truth" that had been missing, enabling consistent, self-service reporting across the organization. - **Departmental Solutions Migration:** Existing dashboards and reports were rebuilt and enhanced in Looker, starting with marketing and campaign analytics. Beyond immediate reporting needs, the semantic layer represents a strategic unlock: a governed data foundation that positions BARK to integrate AI and natural language interfaces directly with trusted enterprise data. ### Phase 3: Manage the Change Throughout the engagement, the Rittman Analytics team worked alongside BARK's data team, focusing on coaching, skills transfer and establishing best practices to ensure they could independently manage and scale the new platform long-term. **Migration Delivered On Time and On Budget:** The complete platform migration, to BigQuery and Looker, was delivered within a 12-month timeline, on budget, and with zero disruption to business operations. Throughout the transition, BARK's teams maintained full access to analytics capabilities, with no gaps in reporting or data availability. ### Business Benefits Delivered The new analytics platform transformed BARK's data capabilities, empowering its teams and creating a foundation for continued growth. - **Established a Single Source of Truth:** The modernized data platform provides consistent, reliable and democratized data access, enabling teams across Marketing, Finance and Customer Experience to make decisions with confidence from a unified version of the truth. - **Increased Operational Efficiency:** Automating data pipelines and operational reporting eliminated manual bottlenecks, reduced overhead and saved significant time and effort for teams such as the Growth Marketing Team. - **Enabled Proactive, Data-Driven Decisions:** With reliable, insightful dashboards, automated alerts and self-service analytics tools, business units can now monitor performance proactively, identify trends and uncover insights that were previously difficult to surface. - **Built a Scalable Foundation for Growth and AI:** The new cloud-native architecture is designed to scale with the business and supports advanced capabilities, allowing BARK to integrate AI and machine learning models directly into its data environment to fuel future growth. - **Empowered the Internal Data Team:** Through a deeply collaborative partnership and extensive knowledge transfer, BARK's data team is now equipped with the skills, processes and tools to independently manage, maintain and evolve its modern data platform. --- # Carwow Google Cloud Migration | Rittman Analytics Canonical: https://rittmananalytics.com/case-studies/carwow URL: https://rittmananalytics.com/case-studies/carwow Description: How Carwow migrated 1,700 dbt models from Snowflake to BigQuery with Rittman Analytics, Wire and AI-assisted delivery. --- ![Carwow logo](https://rittmananalytics.com/__l5e/assets-v1/14bfc0d6-3816-48d5-b08d-be69535cd13a/carwow-logo.png) # Carwow accelerate their migration to Google Cloud with AI and Rittman Analytics ![Becky Allsop](https://rittmananalytics.com/__l5e/assets-v1/3289bcd9-9332-410a-b243-a43a7f13747c/becky-allsop.png) ## Becky Allsop Director of Analytics & Data Science, Carwow > “I had an update with all of the exec and the leadership team yesterday. They were all absolutely amazed at what's been achieved. They were astounded at what's been done in the timeframes, compared to all of their previous experiences in other businesses doing this type of thing.” ![Carwow car review featuring a blue SUV](https://rittmananalytics.com/__l5e/assets-v1/34616742-f8f1-4d30-8069-4bf503c0e5f1/carwow-image-hero.webp) ### Vital Stats ### Website https://www.carwow.co.uk ### Industry Automotive Marketplace, Online Retail ### Company Size c. 500 employees ### Headquarters London, United Kingdom ### Rittman Analytics Services - Data warehouse migration - dbt consulting & support - Data platform implementation - Wire Framework (AI-assisted delivery) ### Technologies Used - Google BigQuery - Snowflake - dbt - Fivetran - Hightouch - Apache Airflow - Google Cloud - Wire Framework and Claude Code ## The Download Carwow is the UK's largest online car-changing marketplace, helping customers buy new and used cars, sell their existing car and lease vehicles. It operates across the UK, Germany and Spain, with dealers and manufacturers using the platform to reach potential buyers. Behind the marketplace sits a large data estate that supports Carwow's marketing, commercial and finance teams. Its Snowflake warehouse contained around 1,700 dbt models, fed by more than 150 Fivetran connectors. From there, 640 Hightouch syncs pushed data into advertising platforms, CRM systems, Google Sheets and dealer-facing reports. Around 300 columns contained personal data protected by masking policies. In 2026, Carwow decided to consolidate its analytics platform on Google Cloud. The plan was to move the warehouse to Google BigQuery, then build towards a new Looker semantic layer and AI-enabled analytics. The first priority was the migration itself: move every existing model to BigQuery with as little change as possible and as quickly as possible, prove that it produced the same results as before and get the new platform safely into production. ## The Challenge A warehouse migration at this scale brings a set of technical and operational challenges. Every model needs to be translated into the new SQL dialect, rebuilt and validated against the original, with any differences understood before the model can move into production. For Carwow, three factors shaped the approach. **A tight delivery window.** Work began at the end of June 2026, with model migration targeted for completion by mid-September. The migration had to move quickly while Carwow's data engineering team continued to support the existing production platform and ongoing business priorities. **A live and evolving codebase.** Development continued throughout the migration, with around 30 new models added each month alongside regular changes to existing models. The migration process had to account for models changing after translation and keep the BigQuery version aligned with the latest Snowflake code. **Source data arriving in parallel.** Historical and source data was being migrated into BigQuery as model translation progressed. In some cases a translated model could not yet be fully built or tested because a required upstream table had not arrived. The process needed to distinguish quickly between translation issues and temporary data dependencies so work could continue. The standard for completion was deliberately rigorous. Each migrated model had to match its Snowflake equivalent exactly, or have any difference documented and agreed with Carwow. For models whose output was sent beyond the warehouse, including advertising feeds and finance exports, exact equivalence was particularly important: these outputs feed directly into operational systems, so validating them before cutover was a key part of the migration. Free 15-question check ## Is your data warehouse migration *ready* for AI-assisted delivery? A structured self-assessment for data leaders and programme sponsors weighing a migration. Fifteen questions tell you whether this approach may suit your work, what needs resolving first and the most sensible next step. [Assess your migration](https://mark-9o3ijqpd.scoreapp.com/) Free. 15 questions. About 3 minutes. [![Wire dispatching one audit agent per platform layer over a dark navy panel](https://rittmananalytics.com/__l5e/assets-v1/c0e93c5d-2017-4577-8edc-b9b2a11cc504/readiness-panel.png) Example result Readiness Medium Code and data access Equivalence and acceptance Business commitment * * * Next step: move to scoping ](https://mark-9o3ijqpd.scoreapp.com/) ## The Solution Rittman Analytics delivered the migration using Wire, its AI-assisted delivery method for data platform work, following a nine-step process built for migrations of this scale. ![Wire's nine-step lift-and-shift migration process: snapshot and audit, translation, validation and lint, development build, equivalence test, release assembly, review and merge, production build, post-merge verification](https://rittmananalytics.com/__l5e/assets-v1/a48ed670-4343-4dab-882d-7eca223c63c5/lift-and-shift-timeline.webp) **1\. Snapshot and audit.** Wire began by building a complete picture of Carwow's Snowflake environment. It mirrored the dbt project and ran audits across the warehouse, ingestion pipelines, reverse ETL, orchestration and security. The audit identified over 1,700 dbt models to migrate, grouped by complexity, and mapped hundreds of protected data fields and their masking rules to the equivalent controls in BigQuery. A short pilot then took a representative group of models through the full process, from translation to validation, establishing a measured delivery pace for planning at full scale. Wire dispatches one agent per layer run concurrently each layer’s audit is independent of the others Audit agentsone per platform layerin parallel db-object-audit-generatedatabase objects every database and schema every table and view, with size and rows stored procedures and UDFs masking policies and tagged columns the full RBAC graph of roles and grants security-audit-generatethe same connection, different questionssecurity which roles hold which grants which columns carry PII tags Findings determine which IAM bindings and BigQuery policy tags target-setup creates. dbt auditdbt models assigns every model a complexity tier triviallowmediumhighblocked ingestion pipeline agentingestion pipelines details of every connection details of every pipeline reverse ETL agentreverse ETL details of every sync details of every destination …and one agent for every further layereach independent until results are assembled **MCP**  legacy DW server · account · role · warehouse Legacy DW platform the system being migrated from, typically thousands of objects **Databases & schemas** **Tables & views** size and row count **Stored procedures & UDFs** **Masking policies** and their tagged columns **Roles & grants** the full RBAC graph MCP Ingestion pipeline platform connectionspipelines MCP Reverse ETL platform syncsdestinations Migration inventory a single file, a tabular record of every item, one row per object, typically thousands pendingtranslateddeferredmigratedfailed objecttypesize · rowsstatecomplexity tablemigrated— UDFpending— dbt modeldeferredmedium pipelinetranslated— syncfailed— ⋮  one row per object dispatch back-and-forth over MCP assemble 1 2 3 **2\. Inventory and batching.** Wire turned the estate into a structured register covering every model, connector and downstream sync, mapped the dependencies between them, and organised the work into ten self-contained migration waves. This structure let much of the migration run in parallel: models were translated, built and checked for equivalence as soon as their dependencies became available. Parallel AI agents handled the repeatable work of translation, validation, comparison and drift detection, while a senior Rittman Analytics consultant acted as release director, overseeing each wave and making the decisions that needed human judgement. With the estate audited, registered and batched, delivery shifted into a delegated, agentic mode: from this point Wire's AI agents carried out the repeatable stages below in parallel and around the clock, with the human team setting direction, reviewing output and making the release decisions. Tier 01 Release director One human Client communications Rulings and waivers Approval gates Judgment catches Fleet-size and budget decisions Rulings, waivers and approval gates Judgment catches Tier 02 Orchestrating session One Fable-class session Dispatches and monitors lanes Owns the register (single writer) Consolidation and backstop passes Assembles PR evidence Amends process docs the day a ruling lands Dispatch Monitoring and PR evidence Lane 01 Translate wave models Lane 02 Validate and lint Lane 03 Build playground copies Lane 04 Run equivalence battery Lane 05 Verify post-merge Lane 06 Drill count divergence Lane 07 Rebuild sync audit Lane 08 Check branch copies Tier 03 Lane agents 6–12 concurrent Stage-scoped work: translation, validation and evidence, one stage at a time. **3\. Translation.** Wire's translation agents rewrote each model's SQL and configuration for BigQuery using a translation guide agreed with Carwow's engineers, covering the less obvious dialect differences: division by zero, rounding, JSON handling, type coercion and timestamps. Translation ran at scale, with hundreds of models handled simultaneously. By mid-August, more than a thousand model comparisons had been completed without identifying a translation defect. **4\. Validation and linting.** Every translated model had to parse and compile in BigQuery before it could move on. Engagement-specific lint rules caught subtle behavioural changes that would not cause a build failure but could change the resulting data. **5\. Sandbox build.** Translated models were built in a controlled BigQuery sandbox before release, with cost checks before larger builds ran. As the migration progressed, the team moved from small batches to building hundreds of models in a single run, with thousands of dbt tests passing alongside them. **6\. Equivalence testing.** Each BigQuery table was compared with the Snowflake table it replaced: row counts, grain, schema and representative value totals, run against production-scale datasets including tables of tens or hundreds of millions of rows. Models were classified as an exact match, a match with understood timing differences, blocked by data availability, or needing further translation work. **7\. Release assembly and pull requests.** Passing models were grouped onto branches in Carwow's own repository and put through final release checks: a full project parse, Carwow's CI, a rebuild against the production dependency graph and a final Snowflake comparison. More than a hundred pull requests were raised into Carwow's repository over the programme, with the majority merged into the production codebase. **8\. Ship, then verify in production.** As the programme progressed, Carwow and Rittman Analytics adopted a production-led release process: models ready to run moved into production rather than waiting for every downstream check to complete. Wire re-ran the data comparisons against the production BigQuery tables after each merge, and Carwow's engineers took a direct role in reviewing and merging pull requests, reducing hand-offs and maintaining pace. **9\. Drift detection and acceptance.** Because the Snowflake project kept evolving during the migration, Wire continuously checked whether translated models had drifted from their originals. Acceptance was managed through dated evidence packs listing the models ready for sign-off with supporting evidence in plain language. Throughout the programme, Carwow had a daily view of progress generated from the dbt manifest, migration register and the live state of both repositories. ## Business Benefits Delivered **Live on BigQuery within weeks.** The full estate was translated in a matter of weeks. By early September almost all in-scope models were merged to Carwow's main branch and running in production. Carwow's engineers reviewed and merged large batches of pull requests in a single day at several points, which kept the final stages moving. **Every model checked against Snowflake.** Over a thousand models were verified against their Snowflake equivalents in production before sign-off. Each result and its evidence went into the migration register, so Carwow had proof the new platform matched the old one. **A small team, more output.** A small Rittman Analytics team delivered the migration alongside Carwow's engineers. AI agents did the repeatable translation and validation work, and the people on both sides spent their time on review and release decisions. **Existing data problems fixed on the way through.** The drift and post-merge checks caught subtle differences across the live estate: join behaviour, upstream scoring logic, historical data loads. Each was traced back to its source and fixed through the normal pull request workflow before it reached the new platform. **Standards Carwow's team still uses.** The translation guide, lint rules and source-pinning practices were documented in Carwow's repository, and their engineers now apply the same patterns in day-to-day development. **Straight into the Looker build.** With the core estate on BigQuery, Carwow moved into the next phase: an enterprise data model in Looker, built with the wider data engineering team. Rittman Analytics continued into that phase for continuity. > “We had a really tight turnaround. Awesome collaboration across the teams to achieve this. I think that it's quite miraculous, actually.” Becky Allsop, Director of Analytics & Data Science, Carwow ## Interested? Find Out More! Rittman Analytics is a boutique data analytics consultancy that helps ambitious, digital-native businesses scale-up their approach to data, analytics and generative AI. We’re authorised delivery partners for Google Cloud along with Oracle, Segment, Cube, Dagster, Preset, dbt Labs and Fivetran and are experts at helping you migrate and upgrade your data platform using an approach that’s right for your organisation’s needs, use-cases and budget. If you’re looking for some help and assistance with your data migration needs or would just like to talk shop and share ideas and thoughts on what’s going on in your organisation and the wider data analytics world, contact us now to organise a 100%-free, no-obligation call, we’d love to hear from you! --- # Case Study: Barton Peveril College - AI-Accelerated Analytics | Rittman Analytics Canonical: https://rittmananalytics.com/case-studies/barton-peveril URL: https://rittmananalytics.com/case-studies/barton-peveril Description: Discover how Rittman Analytics and Barton Peveril College pioneered AI-accelerated analytics for UK education, delivering an MVP analytics platform in just 15 working days. --- ![Barton Peveril Sixth Form College logo](https://rittmananalytics.com/lovable-uploads/barton-peveril-logo.png) # Barton Peveril College Uses Rittman Analytics To Pioneer AI-Accelerated Analytics for UK Education ![Chris Loveday](https://rittmananalytics.com/lovable-uploads/barton-peveril-chris-loveday.jpeg) #### Chris Loveday Vice Principal, Barton Peveril College > "This MVP has proven what's possible when strong partnerships meet a clear vision, high-quality data, trusted metrics, and genuinely accessible analytics. I'm incredibly excited about the next phases of this work and the potential impact it can have on how data is used across education, not just for reporting, but for insight, foresight, and better outcomes for learners and institutions alike." ![Barton Peveril students in science lab](https://rittmananalytics.com/lovable-uploads/barton-peveril-science-lab.png) ### Vital Stats #### Website barton-peveril.ac.uk #### Industry Education #### Company Size 250-500 employees #### Headquarters Eastleigh, United Kingdom ### Rittman Analytics Services • AI-Augmented Analytics Delivery • Data Platform Development • Semantic Layer & Dashboard Design • Training & Enablement ### Technologies Used • Google BigQuery • dbt • Looker • Claude Code • Google Gemini • ElevenLabs ## The Download Barton Peveril Sixth Form College is one of the UK's leading providers of post-16 education, consistently rated Ofsted Outstanding across all key judgements. Serving nearly 5,000 students from across Hampshire, the college has built a reputation for academic excellence, with value-added performance in the top 10-20% nationally and 75% of students progressing to university degrees. From its origins as a pupil teachers' centre in 1904 to its current status as a flagship sixth form college, Barton Peveril has always embraced innovation in pursuit of better student outcomes. This forward-thinking approach led the college to become the UK's first "Google Gemini Academy" in 2024, equipping every teacher with AI-powered tools. But the leadership team recognised that true data-driven decision making required more than modern technology, it demanded a fundamental shift in how performance data was accessed, analysed, and acted upon. ## The Challenge Barton Peveril Sixth Form College had data, but it was fragmented across multiple systems, ProSolution MIS for student records, ALPS for external benchmarking, and various PDF reports for equity analysis. Leadership needed consolidated insights to support strategic decision-making, but building a traditional analytics platform would take months the college couldn't afford. With a director demonstration scheduled for early January and just 19 working days available, Barton Peveril needed a partner who could deliver enterprise-quality analytics at startup speed. They chose Rittman Analytics to help them build a modern data platform that would serve as a strategic asset for years to come. ## The Solution: AI-Augmented Process, Human-Led Judgment At Rittman Analytics, we use AI as a co-pilot, not an autopilot, to accelerate data transformation while keeping expert teams in complete control. Our approach uses AI to automate repetitive tasks and accelerate feedback loops, freeing teams to focus on what matters most: complex business logic and stakeholder collaboration. The Barton Peveril Analytics Platform project showcased this full AI-accelerated methodology. ![AI-Accelerated MVP Development Timeline showing 4-week progression from design to build to MVP rollout](https://rittmananalytics.com/lovable-uploads/barton-peveril-timeline.png) [Learn More About AI-Augmented Delivery →](https://rittmananalytics.com/services/ai-augmented-delivery) ### Accelerated Prototyping Within hours of the initial discovery session, AI-generated dashboard mockups were in stakeholder hands, not as finished products, but as conversation starters. Over an intensive discovery day, we conducted requirements workshops with Directors, Subject Leaders, and Senior Leadership to understand their specific analytical questions. We then used AI to generate interactive dashboard mockups directly from those requirements, making stakeholder conversations concrete and productive from day one. Each mockup was cross-referenced against actual source data DDLs. AI produced a deliverability analysis identifying which visualisations could be built with available data and which required additional sources, surfacing gaps before development began, not after. ### Governance at Speed Every AI-generated artifact passed through explicit human-led review gates. We maintained 100% explainability, semantic clarity, and trust throughout delivery. Together with Barton Peveril, we designed and implemented a dimensional data warehouse in Google BigQuery, a star schema architecture optimised for analytical queries. AI (Claude Code) accelerated the generation of dbt models, staging views, and warehouse tables, while human engineers validated business logic, reviewed JOIN patterns, and ensured data quality. To ensure trust in the platform, we implemented 107 automated data quality tests covering primary keys, foreign keys, referential integrity, and business rule validation. AI generated the test definitions; human reviewers confirmed coverage was complete. Documentation coverage was enforced at 100% across all models, every column, every business rule, fully explainable and auditable. ### Faster Iteration AI instantly generated boilerplate code, documentation, and tests, allowing engineers to focus on solving unique business problems, not repetitive tasks. With the warehouse in place, we extended the platform to include a complete Looker semantic layer and eight production dashboards organised by audience (SLT, Course Directors, ALPS Analysis). AI generated LookML views and dashboard definitions from specifications in a single session; work that would traditionally take weeks was completed in hours. For ALPS benchmarking data locked in PDF reports, Google Gemini 2.5 Pro provided structured extraction using native document vision. When extraction accuracy proved insufficient for production use, the team pivoted to spreadsheet sources, AI accelerated the exploration; human judgment made the call. ### Deeper Collaboration With core infrastructure built in days rather than weeks, Barton Peveril's team was freed from waiting on slow builds and manual processes. More time was spent with stakeholders validating insights and less time debugging pipelines. Just three days after kickoff, domain experts were validating enrolment counts and demographic breakdowns against their source system knowledge, not waiting for a build to complete, but actively testing real data. We introduced Looker's Conversational Analytics as a technology preview, then spent four days refining it through iterative testing with the MIS team. Staff tested questions, identified interpretation issues, engineers corrected SQL logic, and agent instructions were refined. The cycle repeated until accuracy met stakeholder expectations. ### AI-Powered Training and Support To ensure successful adoption beyond initial delivery, we provided comprehensive training materials and an AI-powered support system designed to scale with Barton Peveril's needs. Staff can get started with a platform overview presentation introducing the analytics architecture and key features, then progress to video tutorials covering dashboard navigation and the self-service Explore interface for creating ad-hoc reports. For ongoing support, we deployed a custom voice and chat support bot built on ElevenLabs, trained specifically on Barton Peveril's platform. Staff can ask natural language questions about dashboard details, data sources, and common platform queries, getting immediate answers without waiting for MIS team availability. This AI-powered support layer extends the benefits of our AI-augmented approach from development into ongoing operations. ## Business Benefits Delivered Through our AI-augmented engagement with Barton Peveril, Rittman Analytics delivered a range of impactful benefits, including: - **Accelerated delivery** of a complete MVP analytics platform in 15 working days, a timeline that would traditionally require several months - **Comprehensive data quality assurance** through 107 automated tests with 100% documentation coverage, ensuring trusted outputs for senior leadership and governors - **Self-service analytics capability** enabling staff to explore student outcomes through interactive dashboards, natural language queries, and AI-powered support - **Consolidated data foundation** integrating ProSolution student records with ALPS benchmarking data in a governed, maintainable architecture - **Scalable training and support** through video tutorials and an AI-powered chatbot that reduces ongoing support burden while improving staff adoption - **Strategic asset for future growth** with full operational documentation and a platform designed for long-term extensibility Together, the outputs of this engagement provide Barton Peveril with the foundation to leverage data more effectively in supporting student outcomes and institutional decision-making. --- # Case Study: Breakthrough - State-of-the-Art Data Products | Rittman Analytics Canonical: https://rittmananalytics.com/case-studies/breakthrough URL: https://rittmananalytics.com/case-studies/breakthrough Description: Learn how Breakthrough partnered with Rittman Analytics to deliver state-of-the-art data products with dbt and BigQuery for logistics innovation. --- ![Breakthrough logo](https://rittmananalytics.com/lovable-uploads/7a358e38-ac7b-43bf-abb6-5a567aaa802e.png) # Breakthrough uses Rittman Analytics to deliver state-of-the-art data products ![Jeff Lang](https://rittmananalytics.com/lovable-uploads/8017fa1b-309e-4f3c-8d0f-096e2017ba69.png) #### Jeff Lang Vice President of Technology Innovation, Breakthrough > "Rittman Analytics enabled Breakthrough to shortcut our learnings and deliver higher-quality data products" ![Breakthrough logistics and transportation](https://rittmananalytics.com/lovable-uploads/ef6bf3be-4f05-4d3e-a578-ae672e549180.png) ### Vital Stats #### Website www.breakthroughfuel.com #### Industry Logistics and Supply Chain #### Company Size 51-200 employees #### Headquarters Wisconsin, USA ### Rittman Analytics Services • Data Strategy • Data Centralisation • dbt Consulting & Support ### Technologies Used • Google BigQuery • dbt • Cube ## The Download Breakthrough is a leading innovator in energy and transportation management dedicated to creating transparent strategies for the world's leading shippers. By leveraging their solid dataset of over $20 billion in annual freight spend and more than 1.5 billion gallons of fuel annually, Breakthrough uncovers freight optimisation and removes distortion from traditional transportation practices. ## The Challenge Breakthrough was looking for a partner to enable and work alongside their data team as they built out a modern data stack that provides the analytics insights powering Breakthrough's data-driven products and services. ## The Solution After the successful delivery of an initial two-week discovery and planning sprint, Rittman Analytics was then engaged for an initial six-month program of work to help define and then deliver Breakthrough's state-of-the-art data strategy and data architecture. Rittman Analytics, a dbt Preferred Consulting Partner, trained and then worked alongside Breakthrough's own internal data team as they adopted a modern software development process for delivering their analytics content. --- # Case Study: BridgeU - Measuring User Engagement | Rittman Analytics Canonical: https://rittmananalytics.com/case-studies/bridgeu URL: https://rittmananalytics.com/case-studies/bridgeu Description: Learn how BridgeU used Rittman Analytics to measure and understand user engagement for better EdTech outcomes with BigQuery, Segment, and dbt. --- ![BridgeU logo](https://rittmananalytics.com/lovable-uploads/0902357f-ca8e-465e-bd0b-275451cdc9e3.png) # BridgeU Uses Rittman Analytics To Measure And Understand User Engagement ![Stuart Cochran](https://rittmananalytics.com/lovable-uploads/8510d6fe-e42e-440e-b761-130aa36acd68.png) #### Stuart Cochran Chief Technology Officer, BridgeU > "Rittman Analytics Helped Us Increase Our Data Capabilities And Better Understand The Value In Our User Engagement" ![User engagement analytics visualization](https://rittmananalytics.com/lovable-uploads/27519565-1a90-4d6a-a98a-8c98262c1bf6.png) ### Vital Stats #### Website www.bridge-u.com #### Industry EduTech #### Company Size 51-200 employees #### Headquarters London, United Kingdom ### Rittman Analytics Services • Data Strategy • Data Centralisation • Data Analytics • Customer Data Infrastructure • dbt Consulting & Support ### Technologies Used • Google BigQuery • Segment • Fivetran • dbt • Looker ## The Download BridgeU is a pioneering EdTech company dedicated to assisting young individuals globally in unlocking their potential by equipping schools with advanced university and career guidance tools. Their comprehensive platform offers features such as Smart University Matching, Career Planning, and Document Sending, simplifying the intricate process of university applications through data-driven intelligence and efficient task management. Recognized as one of Europe's most innovative EdTech startups by EdTech Europe, BridgeU has successfully secured £6.4m in venture capital from notable investors including Octopus Investments, Seedcamp, Fresco Capital, and Deborah Quazzo. ## The Challenge BridgeU's ability to be proficient with reporting, exploring, and analysing their data is core to their mission, making analytics a part of their competitive advantage. Upgrading those capabilities required improvements in systems, processes, and people and the objectives of: - **Enhancing user engagement insights**, by developing a deeper understanding of user engagement and thereby enabling BridgeU to refine their products and services more effectively. - **Optimizing outcome measurement**, establishing solid methods to measure, report, and analyze the relationship between user engagement with BridgeU's website, email marketing activity and in-person events and the effect they had on successful student outcomes. However and common to most fast-growing organizations BridgeU faced a number of challenges in delivering this initiative, including: - Data being stored in multiple places, making it difficult to combine - Analysis of that data was time-consuming and difficult, leading to lots of time and effort spent on compiling the data, leaving less left over for complex analysis. - Difficulty in ascertaining whether a business question could be answered, because it was difficult to get an overview of what data was available. ## The Solution Rittman Analytics were engaged therefore to help deliver the first phase of this initiative, adding their expertise with behavioral event analytics and the modern data stack to the domain knowledge of BridgeU's team. Over a period of three months Rittman Analytics worked with the BridgeU team to: - Identify the actions that influenced candidates on their journey - Track and count events that occur following those actions being performed - Identify the key stages in the candidate to student journey - Report on movement through the recruitment funnel that those events influenced Using technology from Google Cloud, Segment and dbt Labs the combined team built a centralized, dimensional data warehouse and set of Looker dashboards that helped answer key business questions such as: - How-much influence Bridge-U had on each candidate movement through the funnel - Which actions are the most influential in moving candidates through the funnel - What proportion of the value of each candidate, at each stage in the funnel, should be attributed (in the classic sense) to Bridge-U activity --- # Case Study: ColourPop - Making Sense of Beauty Data | Rittman Analytics Canonical: https://rittmananalytics.com/case-studies/colourpop URL: https://rittmananalytics.com/case-studies/colourpop Description: Discover how ColourPop cosmetics used Rittman Analytics to transform their data strategy with AWS Redshift and Looker for better beauty insights. --- ![ColourPop logo](https://rittmananalytics.com/lovable-uploads/966c8a62-6107-4556-81b9-1b4735e7a375.png) # ColourPop uses Rittman Analytics to make sense of its data ![Nate Dierks](https://rittmananalytics.com/lovable-uploads/73a8a1de-07cf-4ec9-a4b6-c7d88598f867.png) #### Nate Dierks ColourPop > "Rittman Analytics implementation was absolutely impeccable" ![ColourPop beauty illustration](https://rittmananalytics.com/lovable-uploads/18fe92b1-ac09-4c08-bbf8-256f555ef4b1.png) ### Vital Stats #### Website www.colourpop.com #### Industry eCommerce & Retail #### Company Size 1,001-5,000 employees #### Headquarters Oxnard, CA ### Rittman Analytics Services • Data Strategy • Data Centralisation • Data Analytics • RA\_Warehouse ### Technologies Used • AWS Redshift • Looker ## The Download Colourpop cosmetics was founded in 2014 with a mission to redefine luxury products in the beauty space at affordable prices. From formulating, to producing, manufacturing, and selling, the company interacts directly with cosmetic consumers through its e-commerce platform. ## The Challenge Colourpop came to Rittman Analytics with a data warehouse project. As an operation with a lean team, the founders knew they'd have technology, data quality, and adoption challenges which would make the project stretch out over many months, alongside a limited capacity. ## The Solution In the clients' own words: "Rittman Analytics were the perfect partners- they provided excellent direction on recommendations for technology that fit our organisation, needs, and budgets and their implementation was absolutely impeccable. All work was well defined and expertly documented- I wish all of my vendors worked like Rittman Analytics." --- # Case Study: Conversion Interactive Agency - Upgrading Embedded Analytics | Rittman Analytics Canonical: https://rittmananalytics.com/case-studies/conversion-interactive URL: https://rittmananalytics.com/case-studies/conversion-interactive Description: Discover how Conversion Interactive Agency upgraded their embedded analytics with Rittman Analytics to deliver better client insights and reporting. --- ![Conversion Interactive Agency logo](https://rittmananalytics.com/lovable-uploads/1b6b39c9-eada-4b10-9cb8-196748d0a2e9.png) # Conversion Interactive Agency Uses Rittman Analytics To Upgrade Its Embedded Analytics ![Kelley Walkup](https://rittmananalytics.com/lovable-uploads/996449dc-a5d1-4386-9ce8-31b8ffbb073f.png) #### Kelley Walkup President and CEO, Conversion Interactive Agency > "Rittman Analytics' deep technical expertise with the modern data stack increased our data team's capabilities" ![Conversion Interactive Agency illustration](https://rittmananalytics.com/lovable-uploads/0d75a221-3dff-4ece-9bb5-f3fed6f1b84a.png) ### Vital Stats #### Website www.conversionia.com #### Industry Marketing, Advertising, Transportation #### Company Size 50-200 employees #### Headquarters Tennessee, USA ### Rittman Analytics Services • Data strategy • Data centralisation • dbt consulting & support ### Technologies Used • dbt • Google BigQuery • Looker Embedded • Rudderstack ## The Download Conversion Interactive Agency is a full-service ad agency providing innovative recruiting, retention & training strategies for the transportation industry. Based in Brentwood, Tennessee, Conversion Interactive Agency provide innovative driver recruiting and retention strategies focused on delivering the results carriers need to reach their goals for over 30 years. ## Scalable Infrastructure For Embedded Analytics Conversion Interactive Agency were looking for a specialist partner to help design and build, alongside their own data team, a world-class and scalable data analytics infrastructure to support embedded analytics used in their LeadLink 2.0 platform. LeadLink 2.0 from Conversion Interactive Agency allows carriers to follow the data from the initial driver click on their ad to the hire and beyond. With this data, they can make informed decisions about their recruitment marketing strategy, take action on recurring driver issues, and optimize recruitment and retention efforts for better results. ![LeadLink 2.0 data architecture diagram](https://rittmananalytics.com/assets/conversion-leadlink-architecture-i9paMiWL.png) Over a three-month period Rittman Analytics worked with Conversion Interactive Agency to first define a data strategy and delivery roadmap to deliver their needs, and worked with their data team to prototype a modern data stack based around technologies from Rudderstack, Google Cloud Platform and dbt Labs. ## Composable Customer Data Platforms From Rittman Analytics Following the success of the initial prototype, Rittman Analytics worked with Conversion Interactive Agency to increase their team's analytics capabilities and implement embedded analytics within LeadLink 2.0, giving users of this product market-leading capabilities such as: ![LeadLink 2.0 analytics dashboard](https://rittmananalytics.com/assets/conversion-leadlink-dashboard-DUTdabcS.png) - Being able to drill down into the metrics for each stage of the driver lead funnel - Awareness, Consideration and Conversion. - An understanding of which recruiters and fleet managers perform best and how that supports recruiting and retention. - An ability to make informed decisions quickly and accurately while gaining insights into recurring driver issues in real-time. --- # Case Study: Cyanocapture - Jumpstarting Data Journey with Looker | Rittman Analytics Canonical: https://rittmananalytics.com/case-studies/cyanocapture URL: https://rittmananalytics.com/case-studies/cyanocapture Description: Learn how Cyanocapture jumpstarted their data journey with Rittman Analytics and Looker to achieve better business insights and analytics. --- ![Cyanocapture logo](https://rittmananalytics.com/lovable-uploads/f01c3594-5410-4ddc-b569-1498b7c70149.png) # Cyanocapture Jumpstarts its Data Journey with Rittman Analytics and Looker ![Siggi Sigurdsson](https://rittmananalytics.com/lovable-uploads/16ce9d7e-fc70-4c43-af45-ebaba84ce06c.png) #### Siggi Sigurdsson Systems Engineer, Cyanocapture > "Rittman delivered everything possible on their end and showed us what could be achieved with Looker" ![Laboratory scientist working with samples](https://rittmananalytics.com/lovable-uploads/543ebd0d-e105-470f-bcf7-cdffab673a97.png) ### Vital Stats #### Website cyanocapture.com #### Industry Biotechnology #### Company Size 11-50 employees #### Headquarters Oxford, United Kingdom ### Rittman Analytics Services • Looker Jumpstart ### Technologies Used • Looker • Google BigQuery ## The Download CyanoCapture is developing a groundbreaking solution to fight climate change by improving nature's own carbon-absorbing process in tiny organisms. Winners of the Elon Musk Foundation XPRIZE in 2021, CyanoCapture has created special bacteria that can capture carbon dioxide faster than ever before and turn it into valuable products for industry. This innovative approach helps clean up emissions from factories and power plants while also making useful materials. By making carbon capture profitable, CyanoCapture is creating a win-win solution for both the environment and businesses. To support and enable growth in their business, CyanoCapture now had a need for a scalable business intelligence (BI) solution that would enable them to monitor system health, analyse operational data and communicate results to both internal stakeholders and external investors. ## The Challenge While CyanoCapture's technical team was comfortable working with raw data and command-line tools for their own ad-hoc analysis, they needed a more interactive, intuitive and accessible approach for the wider sharing of results and insights with stakeholders and investors. The company operates a suite of IoT devices that track system inputs, processes, and outputs, generating data every few seconds. This high-frequency data, combined with their need to forecast profitability in the carbon credit market, presented both technical and architectural challenges. The tools that they were familiar with were soon going to run into scalability issues & enriching with third party data was difficult. CyanoCapture had identified Looker as their preferred platform based on its innovative and powerful universal semantic layer and the self-service, low-latency access it provides to CyanoCapture's detailed, granular IoT data stored in Google BigQuery and Google Cloud. ## Solution Delivered Rittman Analytics delivered our Looker Jumpstart Embed consulting package over four weeks, with an objective of getting CyanoCapture up-and-running with Looker and their first embedded use-case successfully-delivered. **Co-Development and Training:** We worked closely with CyanoCapture's technical developers and business users through a series of co-development sessions. These included targeted, hands-on training covering everything they needed to get up to get their real-time data visualised in Looker, from platform configuration to LookML development, and visualisations to alerting. **SME Support:** Throughout the engagement, CyanoCapture had access to one of our Looker experts to support their team, demonstrating advanced features of the platform and rapidly accelerating their confidence and ability to implement those and other features. **Data Architecture and Schema Design:** Cyanocapture also needed help designing the architecture and schema of the data ingestion pipeline. Using some of the available independent development time, we advised on the use of additional GCP services suitable to their IoT data needs and designed a suitable Google BigQuery schema that met their immediate needs and would scale with future plant expansion. ## Benefits Delivered By the end of the Looker Jumpstart engagement, the team had produced a stunning set of dashboards that displayed key information about their plant's performance, operational status and economic outlook. CyanoCapture's team has also developed a strong understanding of the Looker platform and has the structure in place to grow and scale their BI capability. The team proudly showcased their new data platform at a major climate-tech festival, highlighting its role in advancing their mission of carbon capture innovation. --- # Case Study: Florence - Smarter Data-Led Decisions | Rittman Analytics Canonical: https://rittmananalytics.com/case-studies/florence URL: https://rittmananalytics.com/case-studies/florence Description: Discover how Florence used Rittman Analytics to make smarter data-led decisions and develop actionable insights for better business outcomes. --- ![Florence logo](https://rittmananalytics.com/lovable-uploads/48ad20ca-3b89-4f8a-8cfc-7e91a53a6548.png) # Florence uses Rittman Analytics to make smarter data-led decisions ![Charles Armitage](https://rittmananalytics.com/lovable-uploads/3b202ed2-a292-42b3-933a-baca808f1f89.png) #### Charles Armitage Managing Director, Florence > "Rittman Analytics really helped us to develop actionable insights that drive our decision-making" ![Florence healthcare marketplace](https://rittmananalytics.com/lovable-uploads/c980dbf6-7d9c-448b-a5bb-bce964384d5b.png) ### Vital Stats #### Website www.florenceapp.co.uk #### Industry Healthcare #### Company Size 11-50 employees #### Headquarters London, England ### Rittman Analytics Services • Operational Analytics • Data Strategy • Data Centralisation • Data Analytics • dbt Consulting & Support ### Technologies Used • Google BigQuery • dbt • Looker ## The Download Florence is a marketplace that makes social care outstanding through innovation and technology. It uses technology to help healthcare professionals find shifts near them, as well as healthcare organisations, fill rota gaps. ## The Challenge Florence was looking to use data to assess the balance of liquidity within its marketplace and improve the efficiency of user onboarding. ## The Solution Rittman Analytics helped extract data from their platform, blend it with external data sources and draw actionable insights that Florence now uses to drive decision-making. In the client's own words, "The project we did with Rittman Analytics allowed us to achieve our stage 1 BI goals quickly and accurately. I was impressed and surprised by how fast we were able to start getting useful insights from our data." --- # Case Study: FSG Smart Buildings - Smart Building Analytics | Rittman Analytics Canonical: https://rittmananalytics.com/case-studies/fsg-smart-buildings URL: https://rittmananalytics.com/case-studies/fsg-smart-buildings Description: Learn how FSG Smart Buildings partnered with Rittman Analytics to develop advanced smart building analytics and IoT data insights. --- ![Facility Solutions Group logo](https://rittmananalytics.com/lovable-uploads/6691d3f4-b0f5-4677-8012-ff28eb0e3da1.png) # FSG Smart Buildings Uses Rittman Analytics To Transform Facility Management With Advanced Analytics ![Adam Haynes](https://rittmananalytics.com/lovable-uploads/cf16b9a1-8d4c-4ddb-b9ba-93313c9c18d4.png) #### Adam Haynes Chief Product Officer - Head of Product & Development, Facility Solutions Group > "Rittman Analytics was able to do what seemed impossible. With only hours of spin-up they dove head-first into the work with our team at FSG. Their incredible team brought structure and strategy to the newly developing discipline of data analytics within FSG. When they joined, we were 6 weeks late on our internal timelines but within weeks, they were able to make up for the lost time and more." ![Smart buildings technology illustration](https://rittmananalytics.com/lovable-uploads/f9f2bcaf-6117-4eca-bb96-5e1902570ce0.png) ### Vital Stats #### Website http://fsg.com/ #### Industry SaaS & Mobile #### Company Size 2500+ employees #### Headquarters Austin, Texas ### Rittman Analytics Services • Data Centralisation • Embeddable Consulting • Data Enablement • dbt Consulting & Support ### Technologies Used • Google BigQuery • Cube • Embeddable • dbt ## The Download Facility Solutions Group (FSG) is a leading provider of lighting, electrical, and energy management products and services, headquartered in Austin, Texas. Founded in 1982, FSG has grown to become a top 20 electrical contractor in the United States, with six divisions, 42 branches, four regional distribution centers, and 2,300 employees serving customers across all 50 states, Canada, and Puerto Rico. Their divisions include Construction, Lighting, Electrical, Technology, Signs and FSG's Smart Buildings division, established in 1998, that focuses on leveraging technology to optimize building operations and energy management. ## The Challenge In line with their commitment to innovation and service excellence, FSG had embarked on a Smart Buildings initiative with the objective of leveraging data to optimize their operations and provide superior services to their clients. Until now, FSG clients and in-particular those in the convenience store sector had needed skills with spreadsheets and data extracts to be able to act upon the data provided by FSG. Moreover, this data was fragmented and limited their clients' ability to gain a holistic view of business operations, tenant behavior, and financial performance. FSG's team had been directed to make the data they collected and generated more actionable and easier to consume for their clients, and a key aspect of this initiative was the development of a Proof of Concept (POC) for surfacing key performance indicators (KPIs) and reporting tables in their client-facing application. FSG faced several challenges in this endeavour, including a lack of initial experience with new tools, a need to innovate quickly, a desire for perfection, an absence of a roadmap or procedures for the team, the need to ensure data quality and bug-free releases, tool overload, and poor development experience due to team silos. Embeddable, a powerful embedded analytics platform designed to integrate cleanly into applications, had been selected as one of the key enabling technologies for this initiative. ## The Solution As one of the founding members of the Embeddable Partner Program, Rittman Analytics were engaged by FSG to lead the technical implementation of Embeddable and guide FSG's team through the process of delivering these embedded application analytics. The technical solution involved creating Embeddable KPIs that were initially placeholder measures that could be surfaced in advance of their specific business logic being agreed. These placeholder KPIs could then be embedded in a prototype version of the customer-facing application so that all necessary KPIs were available before any client design workshops. Once design was complete, business logic was added to each of the placeholder KPIs followed by user acceptance testing to ensure their accuracy and usability. Rittman Analytics collaborated with Embeddable to gather and confirm requirements for the various KPIs and define a development roadmap for integrating Embeddable technology into FSG's client-facing application. FSG and Rittman Analytics' teams worked-collaboratively to model and warehouse FSG's high-volume smart buildings event data and define a rich, descriptive business semantic layer over the warehoused data. Finally and after conducting thorough testing to ensure KPI accuracy and usability, FSG's data pipeline was then orchestrated and scheduled using Dagster Cloud so that client data could be up-to-date as possible, ensuring that decisions would always be made on up-to-date buildings data. ## Business Benefits Delivered Rittman Analytics played a crucial role in not only implementing the technical solution but also in guiding FSG through the process. The project significantly improved the way FSG's clients interact with the data provided, making it more accessible and actionable. Rittman Analytics' team were also able to lower the cost of operations down for FSG by employing advance data transformations, partitioning and by taking advantage of all the features offered by BigQuery. The integration of disparate data systems provided a holistic view of business operations, tenant behavior, and financial performance, enhancing operational efficiency and enabling more effective buildings management. --- # Case Study: Insighta - Building Multi-Tenant Marketing Analytics | Rittman Analytics Canonical: https://rittmananalytics.com/case-studies/insighta URL: https://rittmananalytics.com/case-studies/insighta Description: Learn how Rittman Analytics helped Insighta build the foundations of their multi-tenant marketing analytics application using modern data stack technologies. --- ![Insighta logo](https://rittmananalytics.com/lovable-uploads/dc3bd3b8-cc99-41d1-ab62-6f8502012b06.png) # Insighta Partner with Rittman Analytics to Build the Foundations of their Multi-Tenant Marketing Analytics Application ![Matthiew Liu](https://rittmananalytics.com/lovable-uploads/995cc2fd-5483-41aa-8178-2d4d5a3f709a.png) #### Matthiew Liu Co-Founder, Insighta > "With a background in building multi-tenant solutions for clients in the marketing technology industry, Rittman Analytics were THE IDEAL partner to help Insighta make our product vision a reality." ![Marketing analytics dashboard](https://rittmananalytics.com/lovable-uploads/3475894d-51f8-4651-9879-a3d279941033.png) ### Vital Stats #### Website https://www.insighta.io #### Industry Marketing Technology, Software Development #### Company Size 1-10 employees #### Headquarters Lehi, Utah ### Rittman Analytics Services • Data centralisation • Marketing attribution • Data enablement • dbt consulting & support ### Technologies Used • Snowflake • Dagster • dbt • Fivetran ## The Download Insighta is a modern technology company focused on empowering businesses with comprehensive and precise marketing attribution insights. Their flagship application provides marketers with a unified view of attribution data, allowing them to track, analyse, and optimise their campaigns effectively. Insighta aims to solve a prevalent issue in marketing analytics, the fragmentation and inconsistencies caused by siloed data from multiple platforms like Facebook, Google, and Pinterest. By consolidating and standardising this data, Insighta enables marketers to make more informed, data-driven decisions. Founded by a team of experienced marketing and data professionals, Insighta combines deep domain expertise with advanced technologies such as predictive analytics, machine learning, and dynamic comparison models. Their core product allows users to apply attribution models such as linear touch, time decay and first action or even use Insighta's proprietary approaches, all in a plug-and-play fashion. The product's user-centric design and rapid deployment capability have made it leader in the competitive marketing attribution technology landscape. ![Insighta platform features and capabilities](https://rittmananalytics.com/assets/insighta-features-Bzni6RRf.png) ## The Challenge Insighta faced several hurdles while building the core functionality of their marketing attribution application. The primary challenge lay in designing a scalable multi-tenancy architecture that could support multiple clients with differing data sources and requirements. Traditional approaches to multi-tenancy lacked the flexibility and efficiency Insighta needed, especially given the diverse formats and structures of their clients' data. Additionally, Insighta needed to ensure that their application could cleanly integrate with a variety of marketing platforms and data sources. Achieving standardisation across disparate datasets, while maintaining accuracy and flexibility, was a complex undertaking. They also sought to incorporate predictive analytics and machine learning to provide actionable insights into future customer behaviours and campaign performance. Compounding these technical challenges was the need for rapid deployment. Insighta required a solution that not only solved these foundational issues but also allowed for the quick onboarding of new clients without disrupting ongoing operations. ## The Solution Rittman Analytics worked closely with Insighta to deliver a comprehensive solution addressing their technical and operational challenges. The project began with the design and implementation of a scalable, mono-repo architecture to support multi-tenancy. Using advanced techniques such as factory patterns and dynamic runtime variable injection, the team created a system that allowed for efficient configuration management and ensured that each client's data was handled uniquely while adhering to a common structure. ![Insighta multi-tenant architecture diagram](https://rittmananalytics.com/assets/insighta-architecture-BnUkrX8-.webp) Key components of the solution included: - **Multi-Tenancy with dbt and Dagster:** The solution leveraged dbt for data transformations and Dagster for orchestration. A centralised mono-repo structure allowed for dynamic parsing and configuration based on client-specific parameters stored in Supabase. - **Customised Conforming Layers:** Rittman Analytics designed a flexible conforming layer that standardised data ingestion while accommodating the unique requirements of each client. This ensured seamless integration of disparate data sources like Shopify, NetSuite, and custom marketing platforms. - **Predictive Analytics and Machine Learning:** The solution integrated machine learning workflows to predict future customer values and campaign performance. These workflows were tightly coupled with dbt transformations to deliver actionable insights. - **Rapid Deployment and Automation:** Automation was built into the solution, enabling faster onboarding of new clients and reducing manual overhead. Pull requests and automated testing processes ensured that updates could be deployed quickly without risking system stability. ## Business Benefits Delivered The collaboration between Insighta and Rittman Analytics resulted in a highly scalable and efficient marketing attribution application. The benefits to Insighta included: - **Enhanced Scalability:** The multi-tenancy architecture enabled Insighta to onboard new clients quickly and efficiently, significantly reducing the time-to-market for their services. - **Operational Efficiency:** Automation and standardisation streamlined data integration and transformation processes, freeing up Insighta's resources to focus on innovation and client engagement. - **Improved Accuracy and Insights:** The solid conforming layer and machine learning workflows ensured that marketing data was accurate, consistent, and actionable. Insighta's clients could now trust their attribution models to make better decisions. - **Future-Proofed Architecture:** The solution's modular and flexible design positions Insighta to easily incorporate new features and adapt to evolving client needs. - **Accelerated Growth:** By addressing foundational challenges and enabling rapid deployment, Insighta was able to focus on scaling their business and expanding their market presence. --- # Case Study: INTO University Partnerships - Maturing Analytics Capability | Rittman Analytics Canonical: https://rittmananalytics.com/case-studies/into-university-partnerships URL: https://rittmananalytics.com/case-studies/into-university-partnerships Description: Learn how INTO University Partnerships matured their analytics capability with Rittman Analytics to deliver better data-driven decisions in higher education. --- ![INTO University Partnerships logo](https://rittmananalytics.com/lovable-uploads/c2badf5e-fcab-4051-8411-3f3e3899e039.png) # INTO University Partnerships Uses Rittman Analytics To Mature Their Analytics Capability ![Julian Ford](https://rittmananalytics.com/lovable-uploads/25cd54f2-79b2-4185-88e5-3efebe4d158c.png) #### Julian Ford VP, Enterprise Information, INTO University Partnerships > "Critically, Rittman Analytics were able to understand INTO's business imperatives quickly" ![INTO team collaboration](https://rittmananalytics.com/lovable-uploads/1486b2fd-ebf2-4281-9b94-b0b9af04d35e.png) ### Vital Stats #### Website www.intoglobal.com #### Industry EduTech #### Company Size 1,001-5,000 employees #### Headquarters Brighton, United Kingdom ### Rittman Analytics Services • Marketing Attribution • Data Strategy • Data Centralisation • Product Analytics ### Technologies Used • Azure • PowerBI • Heap ## Background INTO University Partnerships Limited (INTO) is a global education partnering organisation headquartered in Brighton, United Kingdom. They form long-term joint ventures with globally ranked, ambitious universities in the United States, United Kingdom and Asia. Working with their partners, each year INTO transforms the lives of thousands of students by delivering remarkable academic and cultural experiences. These partnerships enable universities to achieve their internationalisation ambitions and deliver sector-leading performance, exceptional student outcomes and remarkable student experiences. ## The Challenge Driven by a desire to fully understand and optimise its digital transformation projects, INTO needed to baseline its online success metrics and fully understand the net contribution of the channels that contribute to a student application whether that be via the INTOStudy website or class-leading PartnerPortal. ## The Solution INTO selected Rittman Analytics to assist because of the team's depth of experience in enterprise analytics allied to their understanding of leading-edge digital tools and reporting frameworks. Critical for INTO was Rittman Analytics' ability to understand INTO's business imperatives quickly, and then to recommend and articulate a solution to stakeholders across all parts of the business including IT, Marketing and C-Level staff. --- # Case Study: Osprey Charging - Accelerating Operational Reporting | Rittman Analytics Canonical: https://rittmananalytics.com/case-studies/osprey URL: https://rittmananalytics.com/case-studies/osprey Description: Learn how Osprey Charging Network scaled their data platform during rapid EV infrastructure expansion with Rittman Analytics using Snowflake, dbt, and Cube. --- ![Osprey Charging Network logo](https://rittmananalytics.com/lovable-uploads/5370dde2-b6ce-49d8-bd43-875c078c8bc8.png) # Osprey Uses Rittman Analytics To Accelerate Operational Reporting ![Rhys Williams](https://rittmananalytics.com/lovable-uploads/94cec459-a10c-4371-8b23-571b887ed4e9.png) #### Rhys Williams Data Lead, Osprey Charging > "Through their expertise, Rittman Analytics has helped us set up tooling and practises to build a world-class data stack and allowed us to deliver real value to our business." ![Osprey Charging station](https://rittmananalytics.com/lovable-uploads/041b12ac-3d79-46f8-bc37-db60fafa950c.png) ### Vital Stats #### Website www.ospreycharging.co.uk #### Industry Energy & Utilities #### Company Size 51-200 employees #### Headquarters London, United Kingdom ### Rittman Analytics Services • Data strategy • Data centralisation • Data analytics • Data enablement • dbt consulting & support ### Technologies Used • Snowflake • dbt • Fivetran • Cube • Preset • Dagster ## The Download Osprey is one of the largest, most popular and fastest-growing rapid electric vehicle charging networks in the UK. Their mission is to play a leading role in the electrification of transport, ensuring their chargers are thoughtfully placed, reliable and simple to use and accept multiple forms of payment to be truly open-access - and every one is powered by 100% renewable electricity. Osprey fully funds the installation of their chargers and all associated maintenance, partnering with landlords and deploying chargepoints based on the optimum solution for EV drivers in each location. In a UK first, Osprey started rolling-out over 150 high-powered charging hubs across the UK in 2021, bringing dynamic load-balancing to their network. By working with landlords to ensure that each site is expanded in line with demand and ensuring space is optimised at all times, Osprey ensures that the number of chargers available is always ahead of market demand. ## Doubling Network Growth In 2023, Osprey hit the accelerator on growth and installed 800 chargers in just nine months, outpacing other charging networks and continuing to deliver value for landlords and drivers of EVs. Key to this growth is Osprey's use of data and analytics and, alongside this investment in charging infrastructure, Rittman Analytics were engaged to help Osprey scale their data analytics platform and data team capabilities. ## Scaling Data Analytics Capabilities Rittman Analytics worked alongside Osprey Charging's team to deliver three objectives that supported their planned growth: - Upgrade and modernise their data platform to enable the continued scaling of their business - Introduce a modern, agile and best-of-breed analytics delivery workflow for centralizing reporting data sources into a cloud data warehouse - Help form and lead Osprey's data team and data function, ensuring they have the optimal set of capabilities and data team roles Over a period of nine months Rittman Analytics worked alongside Osprey's team to build a centralised, automated reconciliation and reporting process for their business operations team. Using technology from partners at Snowflake, dbt Labs, Preset, Cube and Fivetran and a solid, repeatable analytics delivery workflow, Rittman Analytics led the design and build of a scalable, modern data analytics platform that: - met their immediate reporting needs with solution that could scale as demand from users grew - created processes and a delivery approach embraced by their data team - provided a framework for future expansion of this analytics platform --- # Case Study: Pleo - Modern Financial Data Analytics | Rittman Analytics Canonical: https://rittmananalytics.com/case-studies/pleo URL: https://rittmananalytics.com/case-studies/pleo Description: Learn how Pleo transformed their financial data analytics with Rittman Analytics to deliver better expense management and financial insights. --- ![Pleo logo](https://rittmananalytics.com/lovable-uploads/6a019117-bfd6-4bbe-a430-37863eeb748c.png) # Rittman Analytics and Pleo work hand-in-hand to design and deliver a modern, scalable and user-centric data foundation ![Priscilla Nagashima](https://rittmananalytics.com/lovable-uploads/9723e9f8-dbd0-4b6a-811d-9813de329f85.png) #### Priscilla Nagashima VP of Data, Analytics & AI, Pleo > "Working with Rittman Analytics has been great. They have been a very helpful extension of our own Data team, rolling up their sleeves and building with us. That partnership was key to leveling-up our data foundations, which lets our team focus on what they do best: using data to drive innovation faster." ![Pleo mobile app interface showing business spending platform](https://rittmananalytics.com/lovable-uploads/8bfcc86d-08da-4b08-bcdf-c456ba7222b0.png) ### Vital Stats #### Website https://www.pleo.io/ #### Industry FinTech #### Company Size 501-1,000 employees #### Headquarters Copenhagen, Denmark ### Rittman Analytics Services • Data Centralisation • Data Team Enablement • Data Team Modernisation • dbt Consulting & Support ### Technologies Used • Looker • Google BigQuery • dbt ## The Download Pleo is a trailblazer in the FinTech world, fundamentally changing how businesses think about spending. From real-time financial tracking to advanced treasury tools, Pleo has become one of Europe's leading holistic business spend platforms, at the forefront of empowering businesses to manage their finances more effectively. This isn't just about efficiency; it's about building a culture of trust and autonomy to free up finance teams to focus on strategic work. This powerful, forward-thinking vision has fuelled their unicorn growth and positioned them as a true leader in the future of work. ## The Challenge Pleo's remarkable success and hyper-growth created a new challenge: the data infrastructure that had served them well during their start-up phase needed to evolve to match their future ambitions. To continue innovating and scaling their trust-based platform, Pleo needed to supercharge its data platform and so they sought a partner to help them build this next-generation platform, choosing Rittman Analytics to help turn their data capabilities into a new engine for growth. ## The Solution At Rittman Analytics, we believe that true data transformation and modernisation of your data function is more than a technical project; it's a fundamental change in your "data operating system". Our approach to transforming and modernising data teams and their platforms is built on a proven, three-step methodology that goes beyond surface-level fixes and delivers lasting, sustainable value. This structured approach ensures we not only build the right solutions but also embed the processes and culture required for them to succeed. 1\. Confront the Hard Truth 2\. Create the Change 3\. Manage the Change ### Step 1: Confronting Hard Truths Through Discovery Over an intensive four-week discovery sprint, we embedded ourselves within Pleo and: - conducted in-depth interviews with leadership, domain data leads and engineers. - attended daily stand-ups for both infrastructure and data services teams to observe real-time dynamics. - performed a deep analysis of all existing documentation, from high-level strategy documents to project plans and team-specific notes. Our discovery revealed a central data team navigating a period of change, deeply committed to maintaining core infrastructure, but in-need of support for fast business growth. This highlighted an opportunity to redefine its role as a more visible and influential partner to the business. ### Step 2: Create the Change Following the discovery phase and a shared understanding of the challenges, Rittman Analytics and Pleo worked hand-in-hand to design and deliver a modern, scalable and user-centric data foundation. #### Change #1: Co-Creating the Pleo Analytics Warehouse (PAW) Together, we designed and implemented a new Pleo Analytics Warehouse (PAW) - a structured, multi-layered architecture that moved data from raw ingestion to reliable, business-ready insights. We also introduced a formal five-phase analytics development lifecycle with clear sign-off points, enabling a more consistent, accountable and collaborative approach to building data products. #### Change #2: Streamlining with a Unified dbt-Looker Monorepo To reduce friction and improve delivery speed, we collaborated on the creation of a unified "monorepo" that brought dbt models and Looker assets into a single development environment. This allowed Pleo's data team to work more efficiently across tools, improved the accuracy of releases and introduced a shared QA and testing process that made iteration faster and more reliable. #### Change #3: Expanding Self-Service Through Shared Governance With the new architecture in place, we extended the workflow to include Looker development - enabling domain analysts to build and test their own content with confidence. ### Step 3: Manage the Change Even the most advanced data platform and technically capable team will fall short without the structures and support needed to sustain meaningful change. Our goal was to ensure that new ways of working didn't just take hold, but became embedded in Pleo's culture and operating model. Over time, our role shifted to support and coaching, as Pleo's team grew in confidence and took the lead. ### Business Benefits Delivered Through our engagement with Pleo's data and analytics team, Rittman Analytics delivered a range of impactful business benefits, including: - Accelerated delivery of Pleo's new analytics warehouse, integrating critical data modeling, orchestration and business intelligence capabilities - Increased capabilities of Pleo's internal data team through targeted training and knowledge sharing - Fostered improved collaboration and trust between Pleo's central and distributed data teams, ensuring data outputs were well-aligned with the company's evolving business needs - Provided a more solid, scalable data platform to support Pleo's continued growth Together, the outputs of this engagement will contribute to Pleo's ability to leverage data more effectively to support their business goals. --- # Case Study: Popsa - Clear Views of Marketing and Finance Analytics | Rittman Analytics Canonical: https://rittmananalytics.com/case-studies/popsa URL: https://rittmananalytics.com/case-studies/popsa Description: Discover how Popsa achieved clear views of marketing and finance analytics with Rittman Analytics using AWS Athena, Looker, and Fivetran. --- [Back to Case Studies](https://rittmananalytics.com/case-study) # Popsa uses Rittman Analytics for clear views of marketing and finance analytics ![Popsa](https://rittmananalytics.com/lovable-uploads/0efdef7b-4690-4f31-8c43-0fa67b294f2f.png) Industry: eCommerce, Mobile Apps, Machine Learning, Printing ## Technologies Used ![AWS Athena](https://rittmananalytics.com/lovable-uploads/e3be4e4d-098a-4df0-b810-d93961b99eec.png)AWS Athena ![Looker](https://rittmananalytics.com/lovable-uploads/eec78f99-61e4-4eb3-9d7b-9d5bb4845de8.png)Looker ![Fivetran](https://rittmananalytics.com/lovable-uploads/d50cfde2-3610-4afd-ab0d-1dac8debad57.png)Fivetran > "Rittman Analytics gave us that end-to-end view of customer acquisition cost, revenue and channel effectiveness" ![Tom Cohen](https://rittmananalytics.com/lovable-uploads/f1b1760f-4441-4c10-8c00-4cb925798d0c.png) Tom Cohen CTO, Popsa ### Vital Stats Website:[www.popsa.com](https://www.popsa.com/) Industry:eCommerce, Mobile Apps, Machine Learning, Printing Company size:25 - 50 employees Headquarters:London, UK ### Rittman Analytics Services - Data Strategy - Data Centralisation - Data Analytics ## The Download Popsa's mission is to bring customisable goods to a new audience using the latest technology and the best user experience. Using machine learning technology that puts together the most appropriate photobook layout in just a few tenths of a second, Popsa's mobile app does the hard work so that their customers can focus on the finishing touches. ## Seeking Data Agile and Insights Growth in the business and increasing bottlenecks in providing data to the business had led Popsa to invest in Looker Data Platform as their analytics platform, and now they were looking for a partner to implement Looker and a financial data lake using Amazon Athena, a serverless, interactive query service that queries and analyses data in Amazon S3 using standard SQL. With Christmas fast approaching and Popsa's engineering resource at a premium, Wire delivered an initial set of Finance reports by connecting Looker directly to Popsa's Amazon Athena data lake and leveraged pre-built LookML finance data models from Rittman Analytics for fast project time-to-value. Over subsequent delivery sprints, Rittman Analytics then extended the solution to include marketing and advertising spend, giving Popsa's team an end-to-end view of customer acquisition cost, revenue and channel effectiveness ready in time for their busiest trading period of the year. ![Popsa product showcase](https://rittmananalytics.com/lovable-uploads/c981cc77-e85a-49c5-afc8-5a8c92e95c3b.png) ### Key Outcomes - End-to-end view of customer acquisition cost - Clear revenue and channel effectiveness insights - Fast time-to-value with pre-built LookML models - Ready for peak trading period ## Ready to Get Clear Views of Your Marketing and Finance Data? Let's discuss how we can help you build a unified view of your customer acquisition costs, revenue, and channel effectiveness. --- # Case Study: Qubit - Customer-Facing BI and Data Analytics | Rittman Analytics Canonical: https://rittmananalytics.com/case-studies/qubit URL: https://rittmananalytics.com/case-studies/qubit Description: Learn how Qubit upgraded their customer-facing BI and data analytics with Rittman Analytics for better personalization and customer insights. --- ![Qubit logo](https://rittmananalytics.com/lovable-uploads/e54a1e53-e66e-408c-99a1-cb4e2efc89e5.png) # Qubit uses Rittman Analytics to upgrade its Customer-Facing BI and data analytics ![Henry Porter](https://rittmananalytics.com/lovable-uploads/d43819fa-5ec5-42fb-810a-ab82eef4ec11.png) #### Henry Porter Product Operations, Qubit > "Rittman Analytics possess that deep technical knowledge of Google BigQuery and Looker" ![Mobile analytics and data visualization](https://rittmananalytics.com/lovable-uploads/a8f97ced-740e-49f9-8d8b-8ebd94114745.png) ### Vital Stats #### Website qubit.com #### Industry MarTech #### Company Size 201-500 employees #### Headquarters London, UK ### Rittman Analytics Services • Data Analytics • Data Centralisation ### Technologies Used • Looker • Google BigQuery ## The Download Qubit believes in the power of data to get you closer to your customers, and Qubit Pro is the fastest, most scalable personalisation platform available today. With a deeper understanding of every visitor and all of their interactions, you can deliver targeted experiences that improve long-term customer loyalty, revenue per visitor and customer lifetime value. The highly granular, low-latency behavioural, quantitative and qualitative data Qubit collects for brands is made available for analysis using Qubit Live Tap, accessed through vertical-specific Google BigQuery Business Views and a Looker Source Block for use with the Looker BI tool, the preferred analytics platform for Qubit Live Tap. ## The Ideal Upgrade Partner Qubit had a need to migrate all standard and customised Live Tap Business Views from BigQuery Legacy SQL to Standard SQL and to use this opportunity to implement table partitioning to improve the response time of user queries. As Mark Rittman was the original product manager for Qubit Live Tap Business Views and the Qubit Looker Source Block and is now a Qubit Partner, Rittman Analytics was engaged to carry out the migration and to upgrade the accompanying Looker Source Block. Rittman Analytics carried out the migration efficiently and delivered the improvement in query response time with the introduction of BigQuery table partitioning. --- # Case Study: Rebtel - Becoming More Data-Driven with Modern Analytics | Rittman Analytics Canonical: https://rittmananalytics.com/case-studies/rebtel URL: https://rittmananalytics.com/case-studies/rebtel Description: Learn how Rebtel became more data-driven with Rittman Analytics, implementing modern analytics to improve telecommunications insights and decision-making. --- ![Rebtel logo](https://rittmananalytics.com/lovable-uploads/cfa36688-0e64-4cf8-9c35-1221e2fcc504.png) # Rebtel Uses Rittman Analytics To Help Them Become More Data-Driven ![Bruno Giordani](https://rittmananalytics.com/lovable-uploads/349ea243-2ec6-4bfc-963e-973d4d8edb4f.png) #### Bruno Giordani Senior BI Analyst, Rebtel > "Rittman Analytics enabled us to increase user adoption, scale our analytics capability and become more data-driven." ![Modern communication technology](https://rittmananalytics.com/lovable-uploads/afd24940-21cd-45ba-bc95-1044443d0aa8.png) ### Vital Stats #### Website www.rebtel.com #### Industry FinTech #### Company Size 50-200 employees #### Headquarters Stockholm, Sweden ### Rittman Analytics Services • Data Strategy • Data Centralisation • dbt Consulting & Support ### Technologies Used • Snowflake • Looker • dbt ## About Rebtel Based in the heart of Stockholm and with Balderton Capital and Index Ventures as principal investors, Rebtel is a tech company on a mission to create products serving the 2 billion internationals of the world, starting initially with communications and international calling and now extending to money transfer and financial services. ## The Challenge Rebtel were looking to invest further in their data and analytics capability, migrating existing on-premises legacy data warehouses to Snowflake and using Looker for self-service analytics. They were looking for an experienced implementation partner to lead the initial implementation and help deliver on their wider data analytics goals. Rittman Analytics came on board initially to help get the project moving, reviewing and advising on platform architecture and delivery approach, working with Rebtel's data team on data readiness and delivering the Looker dashboards in order to validate key business metrics and enabling basic recency, frequency and monetary value customer segmentation. ## The Solution Rittman Analytics enabled Rebtel to build a modern, scalable and flexible data platform with version control, automated testing, continuous integration and continuous delivery using technology from Looker, Snowflake, Matillion and dbt. Now as they continued to build out our own internal development capability and extend the usage of Looker throughout Rebtel, they looked forward to working with Rittman Analytics to help us meet their goals of increasing user adoption, scaling their analytics capability and enabling their business operations and product teams in their goals to become more data-driven. --- # Case Study: Rixo - Creating A Single View Of Trading Operations | Rittman Analytics Canonical: https://rittmananalytics.com/case-studies/rixo URL: https://rittmananalytics.com/case-studies/rixo Description: Discover how Rixo fashion brand created a single view of trading operations with Rittman Analytics for better retail and eCommerce insights. --- ![Rixo logo](https://rittmananalytics.com/lovable-uploads/23f6e9be-0892-4034-8836-12007bd76ab8.png) # Rittman Analytics Helps Rixo Create A Single View Of Its Trading Operations ![Orlagh McCloskey](https://rittmananalytics.com/lovable-uploads/7fa4a27a-5e7b-481c-b154-15492d6ab6a2.png) #### Orlagh McCloskey Founder and Brand Director > "Thank you for all your wonderful help on getting us up and running with looker and keeping everyone on board along the way!" ![Rixo storefront](https://rittmananalytics.com/lovable-uploads/66849f19-f816-47f8-8a39-8310bf9543be.png) ### Vital Stats #### Website https://rixolondon.com/ #### Industry Retail, Fashion, eCommerce #### Company Size 11-50 employees #### Headquarters London, United Kingdom ### Rittman Analytics Services • Data strategy • Data centralisation • Data analytics • Data enablement • dbt consulting & support ### Technologies Used • Google BigQuery • Looker • dbt Core and dbt Cloud • Data Virtuality • Stitch ## The Download Founded in 2015 by London College of Fashion graduates Henrietta Rix and Orlagh McCloskey, Rixo is a fashion brand based out of London, England famous for hand-painted prints, premium materials and vintage-inspired silhouettes that counts Taylor Swift, Lily James and the Duchess of Cambridge as fans. Started as a passion project run from the living room of their London flat Rixo had, by 2019, grown quickly into business with an international ecommerce presence, physical stores in London and plans for more together with a wholesale business with two warehouses. ## Rixo's Plans For Data-Driven Growth Rixo now had a need to monitor revenue streams, stock levels, and their retail and ecommerce activity, whilst also having a desire to innovate and improve their business via analysis of the data those activities produced. As a fast-growing startup, Rixo had, however, no in-house analytics capability or tools and relied instead on the limited analytics features that came with the software services that powered their website and retail stores. These in-app analytics were usable for basic monitoring but their lack of cohesion and disconnected nature made making decisions requiring a joined-up view of the business difficult. What Rixo needed was a centralised and joined-up view of their data, organised for decision-making and capable of handling ecommerce-specific data challenges such as: - Linking returns to sales - Reporting gift cards not as revenue, but as payments for subsequent orders - Implementing the 4-5-4 week retail calendar used for like-for-like reporting and then presented to business users as intuitive, self-service dashboards and reports. ## Experts In The Modern Data Stack Rixo partnered with Rittman Analytics, a Google Cloud Partner based in Brighton, United Kingdom with extensive experience delivering modern data stack solutions for high-growth retail and online businesses such as Rixo. Focusing first on meeting Rixo's retail and ecommerce business and key business metrics, such as sales revenue and margin, unit and order volumes, and new and existing customer splits, Rittman Analytics implemented a data analytics solution built on a modular and flexible foundation of Google Cloud and partner data warehousing and data analytics technologies. ![Rixo modern data stack architecture](https://rittmananalytics.com/assets/rixo-stack-B5GiH0oa.webp) Rittman Analytics was able to set up flexible connections to the various data sources using Data Virtuality, store their data in Google BigQuery, and then use dbt Core and dbt Cloud to standardise and transform those sources into a structure optimised to answer business questions. We were also able to adapt this solution as new data sources were added and provide guidance on how best to structure an in-house data team for Rixo. Overall, Rixo's considered this first phase of our engagement to be "a very comprehensive condensation of what was previously a sprawling mess of unused data sources and disconnected software." ## Delivering Rixo's Trade Pack KPIs Rixo's new Data Team is now collaborating with Rittman Analytics on building out an automated Trade Dashboard KPI pack comprising a set of Looker dashboards that provide high-level Trade process KPIs along with more detailed KPIs for specific parts of that process. Every Tuesday, Rixo conducts an extensive analysis of the previous week's performance, focusing on various aspects like online and in-store sales, advertising efficiency, brand development, and wholesale distribution. This process, crucial for strategic decision-making, required a significant manual effort. ![Rixo Trade Dashboard Pack KPI structure](https://rittmananalytics.com/assets/rixo-trade-pack-hxhRT7n-.webp) Around a third of Rixo's head office staff dedicated between half and all of their time on Mondays to compile this data in the form of a slide pack for review. The complexity in this task lay in needing to amalgamate data from diverse sources including Netsuite, Google Analytics 4, Google Search Console, Shopify, Google Ads, Meta Ads, and Ometria. Data in most cases needed manual manipulation and standardisation in Microsoft Excel along with the application of business rules on customer and revenue definitions specific to Rixo. Rittman Analytics partnered with Rixo to enhance their data engineering capabilities and streamline the analytics delivery process. Their role was pivotal in integrating new data pipelines into Rixo's existing data warehouse and combining them with the platform infrastructure developed in the first phase of their engagement. As well as providing guidance for Rixo's data team as they took on management of the delivery program, a significant part of Rittman Analytics' contribution was in developing a clear roadmap for future data connectivity and expansion of the platform, ensuring that new data sources were efficiently integrated and the warehouse was ready for future growth. According to Alex Macartney, Data Analyst at Rixo, "Rittman Analytics's expertise and experience with building out data infrastructure has massively sped up the timeframe in which we are able to implement the Trade Dashboard. Their ways of working and engineering methodologies have set a terrific foundation for the project and have really acted as a force multiplier by enabling the Rixo Data Team to function above their expected level. Furthermore, they have been incredibly generous with their knowledge and are always willing to pass on best practice or tips to the Rixo Data Team" --- # Case Study: ROUVY - Rebuilding Data Infrastructure | Rittman Analytics Canonical: https://rittmananalytics.com/case-studies/rouvy URL: https://rittmananalytics.com/case-studies/rouvy Description: Learn how ROUVY rebuilt their data infrastructure with Rittman Analytics to power their indoor cycling platform with better analytics and user insights. --- ![ROUVY logo](https://rittmananalytics.com/lovable-uploads/bf807715-48cb-49e0-8424-9e6174a03664.png) # ROUVY: Rebuilding Data Infrastructure with Rittman Analytics ![Erik Hegely](https://rittmananalytics.com/lovable-uploads/e06832d7-2732-43a0-84c2-f0cf8bfd3d31.png) #### Erik Hegely Growth Director, ROUVY > "Rittman Analytics was instrumental in designing the architecture of the new data stack and providing senior engineering support." ![ROUVY data infrastructure illustration](https://rittmananalytics.com/lovable-uploads/fff0cbb5-81b6-4066-a8e5-28ff4437ebc8.png) ### Vital Stats #### Website https://rouvy.com/ #### Industry Software Development / Esports #### Company Size 51-200 employees #### Headquarters Prague, Czechia ### Rittman Analytics Services • Data Strategy • Data Centralisation • Data Analytics • Data Enablement • dbt Consulting & Support ### Technologies Used • Dagster • Looker • Fivetran • Airbyte • dbt Core • Snowplow ## The Download ROUVY is the leading realistic indoor cycling app that combines technology and sport to let cyclists ride real video routes from around the world. Offering over 1,300 routes and essential training features, the app is popular among both hobby riders and professional cyclists. Its mission to create an authentic, immersive cycling experience has led to partnerships with global cycling brands like Lidl-Trek, La Vuelta, and Strava. A company with over 100 employees, ROUVY is headquartered in Prague, Czechia, and was recognised as one of the 50 fastest-growing tech companies in Central Europe by Deloitte Technology Fast 50 CE in 2023. ## Legacy Data Stack Before the transformation, ROUVY relied on a mix of legacy systems, including a proprietary transformation engine, GoodData, Airflow, and an initially disconnected Chargebee system used for handling subscription management. These systems struggled to meet the growing demands of ROUVY's platform, compounded by inefficient event tracking and data integration processes. ### Challenges - **Custom Connectors:** Data extraction relied on custom Python code, making updates and additions difficult. - **Transformation Engine Performance Issues:** Struggled with large datasets, causing extended processing times and inconsistent metrics. - **GoodData Limitations:** Memory-intensive operations and limited custom visualisation capabilities slowed data availability. - **Airflow Scalability Issues:** Limited scalability and real-time data processing capabilities. - **External Dependencies:** Heavy reliance on external consultants led to weaker internal documentation and challenging refactoring processes. - **Event Tracking and Data Loss:** Significant gaps in event tracking with Google Analytics, leading to data loss and lack of control over event quality across ROUVY's first-party data. ### Impact - **Delayed Decision-Making:** Slower insights affected business agility. - **Increased Costs:** Frequent reliance on external resources for maintaining and updating BI systems added significant costs. - **Customer Data Insights:** Poor tracking and inconsistent data handling impacted marketing strategies and product engagement. ## Transformation with Rittman Analytics The primary goals were to address performance issues, create a consistent data pipeline, democratise data access, and build a scalable infrastructure using the following data stack: - **Dagster for Data Orchestration:** Enhanced scalability and reduced processing times. - **Looker for BI and Analytics:** Centralised data access, empowering teams with reliable analytics tools. - **Fivetran:** For reliable data handling for critical sources. - **Airbyte:** For cost-effective connecting of multiple sources, including Chargebee. - **dbt Core:** Provided a solid transformation layer, enabling efficient and scalable data transformations. - **Snowplow:** Improved behavioural data collection and event tracking quality. ## Outcome To transform their data analytics capabilities, ROUVY partnered with Rittman Analytics and Snowplow over a six-month period. Rittman Analytics was instrumental in designing the architecture of the new data stack and providing senior engineering support. --- # Case Study: Torticity - Creating A Foundation For In-App Analytic Insights | Rittman Analytics Canonical: https://rittmananalytics.com/case-studies/torticity URL: https://rittmananalytics.com/case-studies/torticity Description: Discover how Torticity built the foundation for in-app analytic insights with Rittman Analytics using Snowflake and dbt for legal technology analytics. --- ![Torticity logo](https://rittmananalytics.com/lovable-uploads/cc8ba288-7b13-4d29-bbd9-bf1c4ec7e276.png) # Torticity Uses Rittman Analytics To Create A Foundation For In-App Analytic Insights ![William Seward](https://rittmananalytics.com/lovable-uploads/d0160de9-bd3e-4bd2-a3d8-84875a26457c.png) #### William Seward Senior Vice President, Technology > "Rittman Analytics increased our data team's capabilities and helped us build a solid foundation for embedded analytics." ![Torticity team](https://rittmananalytics.com/lovable-uploads/f205a606-6ea4-4dcc-85a9-c8701c2d535b.png) ### Vital Stats #### Website www.torticity.com #### Industry Legal Services #### Company Size 100-200 employees #### Headquarters Boca Raton, Florida ### Rittman Analytics Services • Data strategy • Data centralisation • Data analytics • dbt consulting & support ### Technologies Used • Snowflake • dbt • ThoughtSpot ## The Download Torticity is a technology-enabled legal services company that provides outsourced support to law firms operating in the mass tort litigation space. Their services specialize in full end-to-end mass tort case management process, including intake confirmation, medical record retrieval, medical record review, delivery of completed case packets and continued support all the way through settlement. They leverage technology to streamline the process and customize it to fit the specific needs of each type of litigation. Their intake confirmation specialists verify client information and capture the latest treatment providers with highest level of precision prior to initiating the medical retrieval process. During the retrieval stage, the retrieval team oversees submission of HIPAA Authorized requests and ensures timely fulfillment. Their medical record review team reviews medical records to locate relevant findings to qualify or disqualify a case. Finally, their settlement support team assists with completing discovery documentation process including preparation of documents such as the plaintiff fact sheets, and other necessary paperwork required to support the law firms throughout the settlement process. All these services are designed to help law firms scale their business by managing their mass tort cases in the most efficient way. ## Enabling Embedded Analytics Torticity had recently invested in a modern data stack and were looking for a partner to help deliver their goal of providing analytics insights to users via Thoughtspot embedded analytics. Thoughtspot's capabilities depend on a solid dimensional model underneath it, providing a structured ad-hoc query layer that makes accessing curated data so easy for end-users. ## Trusted To Transform Rittman Analytics were engaged initially to increase the capabilities of Torticity's data team, working with their data lead to understand their business process and then model and transform their data in Snowflake Data Warehouse using dbt and Rittman Analytics' delivery workflow. Rittman Analytics' analytics engineers and data architect worked alongside Torticity's Data Team to share their approach and best practices, giving Torticity's team the confidence to regularly externalise data to their customers and increase the value of their mass tort management service. --- # Case Study: Welbee - Bringing AI-Driven Insights to Education | Rittman Analytics Canonical: https://rittmananalytics.com/case-studies/welbee URL: https://rittmananalytics.com/case-studies/welbee Description: Learn how Welbee partnered with Rittman Analytics to bring AI-driven insights to education, improving staff wellbeing analytics for schools. --- ![Welbee logo](https://rittmananalytics.com/lovable-uploads/792050b6-78f5-44d6-a30c-23fd017ac8bd.png) # Bringing AI-Driven Insights to Education: Welbee's Partnership with Rittman Analytics ![Tim Handley](https://rittmananalytics.com/lovable-uploads/965c4cca-4779-459d-a5c7-fe0ff6d06d26.png) #### Tim Handley Chief Product and Technology Officer, Welbee > "Working with Rittman Analytics to develop the AI capability of our new Edu Intelligence platform has been transformative for Welbee and our customers" ![Students in classroom](https://rittmananalytics.com/lovable-uploads/a7120f43-8cbc-4b97-ba12-b085b0b461ab.png) ### Vital Stats #### Website https://welbee.co.uk/ #### Industry EduTech #### Company Size 10-20 employees #### Headquarters Liverpool, United Kingdom ### Rittman Analytics Services • Data Analytics • Data Enablement • Generative AI ### Technologies Used • Google BigQuery • Vertex AI • Google Gemini 1.5 • Langchain • Cloud Functions ## The Download Welbee is a multi-award-winning company offering tools and surveys designed to help schools assess and improve the wellbeing of their staff. Using survey data, Welbee delivers actionable insights that empower schools across the UK to implement strategies that foster thriving educational environments. Using a systematic approach they call "The Welbee Way," they guide schools to measure, act on, and embed wellbeing into their culture. Their new Edu Intelligence platform enables schools to create, run, and analyse surveys for their staff, students and parents, providing a comprehensive view of wellbeing across the whole school community. ## The Challenge When we caught up with the Welbee team, their goal was to deliver the same award-winning insights and recommendations from their Wellbeing Survey within the Edu Intelligence platform. Their staff wellbeing survey was already highly successful, but their process relied on manual analysis and interpretation of survey results, causing delays for schools eager to act on their data. Welbee saw an opportunity to make their data more accessible and actionable for their users. Welbee also wanted to empower schools with more dynamic and interactive ways to explore their data beyond static charts and traditional reporting methods. By leveraging AI, Welbee aimed to create a conversational experience that could uncover hidden trends and deliver actionable, data-driven recommendations for improvement. With an existing Looker platform in place, Welbee sought to integrate this new AI-powered analysis into a chatbot interface. This approach would ensure a seamless and intuitive user experience within a familiar environment. ## The Solution To address Welbee's goals, they partnered with us to develop an AI-powered proof of concept (POC) that integrated AI-driven insights into their Edu Intelligence platform. This POC laid the groundwork for Intelligent Ed, an AI-powered staff wellbeing assistant designed specifically for the education sector. The POC solution was made-up of three core components: - **AI-Driven Pre-Analysis:** This component automated survey data processing using Google BigQuery, and Vertex AI. It leveraged Google's Gemini 1.5 large language model (LLM) to generate insights and highlight differences between subgroups within survey respondents. - **Text-to-SQL Chatbot:** Rittman Analytics used Langchain and Langsmith to develop a text-to-SQL agent that allowed users to query survey data conversationally. The agent interpreted natural language questions, broke them down into relevant components, and generated SQL queries to retrieve data from BigQuery. - **RAG Integration:** This component integrated Welbee's existing RAG (Retrieval-Augmented Generation) system with the chatbot, as part of a wider LangGraph implementation. RAG enhanced the chatbot's responses by retrieving relevant contextual information from external knowledge sources. Built on Google Cloud Platform (GCP) technology, Welbee's survey data was stored and processed in BigQuery while Vertex AI provided the machine learning capabilities for AI-driven pre-analysis and natural language understanding for the chatbot. ## Faster, More Actionable Insights > "Rittman Analytics' expertise in AI and Google Cloud allowed us to help our customers unlock the true potential of their data, delivering faster, more actionable insights. The Rittman team is a pleasure to work with, and they have a great level of knowledge, skill, and expertise in all things data analytics. We are thrilled with the results of this project and look forward to continuing to work with the Rittman team in the future" > \- Tim Handley, Chief Product and Technology Officer, Welbee ## Benefits Delivered The collaborative nature of the project leveraged the combined expertise of Rittman Analytics in AI and Google Cloud Platform with Welbee's in-depth knowledge of their platform and users' needs, delivering the following benefits to the business: - **Faster Time to Insights:** The AI-driven pre-analysis eliminates manual analysis and provides insights immediately upon survey closure, allowing schools to act on their data more quickly, leading to faster improvements in staff well-being. - **Enhanced User Experience:** The Text-to-SQL chatbot enables users to interact with their data conversationally and obtain insights in real-time without needing technical expertise. - **Deeper and More Actionable Insights:** The combination of AI-powered analysis and RAG integration provides richer context and surfaces hidden trends that might be missed with traditional reporting. - **Competitive Advantage:** Intelligent Ed, its underlying concepts proven by this POC, will position Welbee as an innovator in the education technology sector by offering a modern solution that sets them apart from competitors. --- # Technical Briefing - Website Architecture & Technology Stack | Rittman Analytics Canonical: https://rittmananalytics.com/technical-briefing URL: https://rittmananalytics.com/technical-briefing Description: Comprehensive technical briefing on Rittman Analytics website architecture, built with React, TypeScript, Supabase, and modern data technologies. --- ## Executive Summary The Rittman Analytics website is a modern React-based web application built for scalability, content management, and external integrations. It serves as both a marketing platform and a content hub with dynamic capabilities for programmatic content generation and API integrations. ## Core Technologies Stack ### Frontend Framework - **React 18.3.1** with TypeScript for type safety - **Vite** as the build tool and development server - **React Router DOM 6.26.2** for client-side routing - **TanStack React Query 5.56.2** for server state management and caching ### Styling & UI - **Tailwind CSS** with custom design system tokens - **Shadcn UI** component library with custom variants - **Radix UI** primitives for accessibility - **Lucide React** for iconography - **Inter** font family from Google Fonts ### Backend & Database - **Supabase** as the Backend-as-a-Service platform - **PostgreSQL** database with Row Level Security (RLS) - **Supabase Edge Functions** for serverless API endpoints - **Supabase Storage** for file management ### Analytics & Integrations - **Segment Analytics** for event tracking - **Google Analytics** (G-M1P5TSNCVM) - **HubSpot** integration for lead management - **Calendly** widget for meeting scheduling - **ConvertBox** for lead capture ## Database Architecture ### Core Tables 1. **blog\_posts** - Content management for blog articles 2. **podcast\_episodes** - Podcast content with audio file management 3. **case\_studies** - Customer success stories 4. **profiles** - User profile data 5. **podcast\_settings** - RSS feed configuration ### Security Model - Row Level Security (RLS) policies on all tables - Email-based access control for Rittman Analytics team (@rittmananalytics.com) - Public read access for content tables - Authenticated user restrictions for content creation/editing ## Programmatic Content Capabilities ### Current API Integrations The site has several edge functions for automated content management: 1. **RSS Import Function** (`import-rss`) - Automatically imports blog content from external RSS feeds - Processes HTML content and metadata - Stores imported content in the database 2. **Podcast RSS Import** (`import-podcast-rss`) - Imports podcast episodes from RSS feeds - Handles audio file metadata and iTunes-specific fields - Manages episode numbering and categorization 3. **Auto-Tagging System** (`auto-tag-post`) - Uses OpenAI API to automatically generate tags for content - Analyzes content and suggests relevant categories - Supports both individual and bulk tagging operations 4. **Audio File Linking** (`link-audio-files`) - Links podcast episodes to hosted audio files - Manages audio file metadata and URLs 5. **Episode Number Parser** (`parse-episode-numbers`) - Extracts and standardizes episode numbers from titles - Ensures consistent episode numbering across the podcast feed ### External API Integration Patterns #### Edge Function Structure ``` // Standard pattern for external API calls import { serve } from 'https://deno.land/std@0.168.0/http/server.ts' const corsHeaders = { 'Access-Control-Allow-Origin': '*', 'Access-Control-Allow-Headers': 'authorization, x-client-info, apikey, content-type', } serve(async (req) => { try { // Get API key from Supabase secrets const apiKey = Deno.env.get('EXTERNAL_API_KEY') // Make external API call const response = await fetch('https://external-api.com/endpoint', { headers: { 'Authorization': `Bearer ${apiKey}` } }) const data = await response.json() // Process and return data return new Response(JSON.stringify(data), { headers: { ...corsHeaders, 'Content-Type': 'application/json' } }) } catch (error) { return new Response(JSON.stringify({ error: error.message }), { status: 500, headers: { ...corsHeaders, 'Content-Type': 'application/json' } }) } }) ``` #### Secrets Management - API keys stored securely in Supabase Edge Function Secrets - Current secrets include: OpenAI API key, Supabase credentials - New secrets can be added via Supabase dashboard ### Content Generation Capabilities #### AI-Powered Content - OpenAI integration for content enhancement - Automatic tag generation based on content analysis - Potential for AI-generated summaries and descriptions #### RSS and Feed Processing - Automatic content import from multiple RSS sources - Content deduplication and normalization - Metadata extraction and enrichment #### Dynamic Content Features - Podcast RSS feed generation (`/drilltodetail/rss.xml`) - SEO-optimized URL structures for blog posts - Date-based and slug-based routing for content ## Site Structure & Architecture ### Page Organization ``` src/ pages/ Index.tsx # Homepage About.tsx # Company information Services.tsx # Service overview services/ # Individual service pages solutions/ # Industry-specific solutions partners/ # Partner integration pages Blog.tsx # Blog listing BlogPost.tsx # Individual blog posts Podcast.tsx # Podcast listing CaseStudy.tsx # Case studies case-studies/ # Individual case study pages components/ ui/ # Shadcn UI components navigation/ # Navigation components blog/ # Blog-specific components [feature-components] # Feature-specific components services/ analytics.ts # Segment integration calendly.ts # Meeting scheduling ``` ### Navigation Structure - **Services**: Data platform development, team enablement, embedded analytics, technical consulting - **Solutions**: Industry-specific offerings (FinTech, EduTech, Marketing Analytics, etc.) - **Partners**: Technology partner pages (Google Cloud, Cube, dbt Labs, etc.) - **Resources**: Blog, podcast, case studies, methodology ### Routing Configuration - Dynamic routing for blog posts with date-based and slug-based URLs - Catch-all routing for flexible content management - Service redirects for URL consistency ## Development & Deployment ### Development Environment - Local development with Vite dev server - TypeScript for type safety - ESLint for code quality - Hot module replacement for rapid development ### Build Process - Vite for optimized production builds - Asset optimization and minification - TypeScript compilation - Tailwind CSS purging for minimal bundle size ### Deployment Platform - Hosted on Lovable platform - Automatic deployments from code changes - CDN distribution for global performance ## Content Management Capabilities ### Blog Management - Rich text content with HTML support - Image integration with Supabase storage - Tag-based categorization - RSS feed generation - SEO-optimized metadata ### Podcast Management - iTunes-compatible RSS feed generation - Audio file hosting via Supabase storage - Episode metadata management - Automatic episode numbering ### Case Study Management - Structured data for customer stories - Logo and image management - Technology and service tagging - Sponsor quote integration ## Extensibility & Future Capabilities ### API Integration Potential The architecture supports easy addition of: - CRM integrations (Salesforce, HubSpot API) - Marketing automation platforms - Social media APIs - Analytics platforms beyond current integrations - E-commerce platforms - Email marketing services ### Content Automation Opportunities - Automated social media posting - Newsletter generation from blog content - SEO content optimization - Competitive analysis integration - Industry news aggregation - Client portal integrations ### Scalability Considerations - Supabase can handle significant traffic growth - Edge functions provide serverless scalability - CDN integration ensures global performance - Database optimization through proper indexing and RLS policies ## Security & Compliance ### Data Protection - Row Level Security on all database tables - Secure API key management via Supabase secrets - CORS configuration for secure cross-origin requests - Authentication-based content management ### Performance Optimization - React Query for efficient data caching - Lazy loading for code splitting - Optimized images and assets - Minimal bundle sizes through tree shaking ## Monitoring & Analytics ### Current Tracking - Segment Analytics for comprehensive event tracking - Google Analytics for traffic analysis - Calendly event tracking for lead conversion - Console logging for development debugging ### Performance Monitoring - Built-in error boundaries for React error handling - Network request monitoring capabilities - Console log access for debugging This technical architecture provides a solid foundation for scalable content management, external API integrations, and programmatic content generation while maintaining security and performance standards. --- # Data Analytics Project Planning Checklist | Rittman Analytics Canonical: https://rittmananalytics.com/data-analytics-project-checklist URL: https://rittmananalytics.com/data-analytics-project-checklist Description: Complete checklist for planning data analytics projects. Essential activities for scoping, defining, and designing successful data analytics initiatives. Expert guidance from Rittman Analytics. --- The Definitive Guide To Planning Your Data Analytics Initiative Coming up with a project planning checklist for your data analytics project is one of the most challenging tasks, particularly for the uninitiated. To help you, we've compiled a list of 'must-do' activities that we've found are essential to successfully scoping, defining, designing and planning a data analytics project. [See client results ›](https://rittmananalytics.com/case-studies) 45 minutes · no obligation · you'll speak with a founder, not sales ## Get a free checklist toolkit: Project Planning Checklist + Example Business Case Serious about delivering a successful data analytics project? Download the exact data analytics project checklist toolkit we use on client engagements and learn advanced tactics for data analytics project planning. Toolkit includes: ✅ Data Analytics Project Planning Checklist Spreadsheet (for Excel/Google Sheets) ✅ Data Analytics Business Case Preparation Spreadsheet (for Excel/Google Sheets) ✅ Pre-populated Example Data Analytics Business Case (for MS Word/PDF viewer) ![Data Analytics Project Planning Checklist Toolkit](https://rittmananalytics.com/lovable-uploads/8a3c7c6f-0d5d-4dc7-ac08-0c72c835200f.png) ## Data Analytics Project Planning Checklist This checklist is designed to guide you through the initial planning stages of your data analytics project, ensuring you have a comprehensive understanding of the scope and requirements. Rittman Analytics can support you at every step, leveraging our Discovery Framework to guarantee the success of your project. ### 1\. Define Your Objectives Defining clear, actionable objectives at the outset is crucial for the success of any data analytics project. Objectives should align with your business goals and address specific challenges or opportunities. Rittman Analytics' strategy workshops expedite this process, helping to capture business outcomes and value propositions efficiently. ☐ Start by clarifying what you hope to achieve with your data analytics project. ☐ Identify the key business questions you want to answer and the decisions you aim to inform. ☐ Define clear, measurable objectives that align with the project's overall vision and goals. ### 2\. Assess Your Data Understanding the state of your current data assets is foundational to planning your data analytics project. This involves evaluating the quality, volume, variety, and veracity of your data to ensure it can support your objectives. Rittman Analytics specializes in transforming data landscapes into scalable, trusted platforms, starting with a thorough data assessment. ☐ Take inventory of your available data sources and assess their quality and relevance to your objectives. ☐ Review and audit the current state of your data, including any limitations or challenges working with it. ☐ Determine your data extraction and data replication needs, including replication frequency and expected data volumes. ### 3\. Understand Your Technology Stack Reviewing and understanding your current technology stack is critical to identifying potential gaps that may hinder your project. This step ensures you have the right tools for data storage, processing, analysis, and visualization. ☐ Review the technologies currently in use for data storage, processing, and analysis. ☐ Identify any gaps in your technology stack that could hinder your project's success. ☐ Evaluate and choose a cloud data warehouse that suits the use-cases and types of data that your project will need to handle. ### 4\. Plan for Data Governance Plan for and understand the importance of establishing clear policies and roles for data management to ensure security, quality, and compliance. ☐ Establish clear data governance policies to ensure your data remains secure, compliant, and of high quality. ☐ Define roles and responsibilities around data management and usage. ☐ Create a governance framework that supports scalable and trusted data platforms. ### 5\. Design Your Data Model & Data Flows An effective data model is foundational to any analytics project. It should be designed to support your objectives, ensuring performance, scalability, and data quality. Rittman Analytics' Discovery Framework includes a design phase focused on creating an optimal information architecture that meets these criteria. ☐ Create an initial logical entity model that defines key activities, actors and actions within the scope of your project. ☐ Design a physical data model for your chosen cloud data warehouse that consider scalability and performance from the start. ☐ Map-out your planned data flows, data transformations and define your testing strategy. ### 6\. Set Up Your Analytics Team Building a capable analytics team is critical to the success of your data analytics project. This team should possess a mix of technical, analytical, and business skills to cover the entire spectrum of data analytics functions, from data engineering and data science to business analysis and project management ☐ Identify the roles and expertise required to deliver your project. ☐ Assess your team's current capabilities and identify any skill gaps. ☐ Find a delivery partner who can provide access to specialized skills and experiences and accelerating your project's success. ### 7\. Develop a Project Roadmap A detailed project roadmap is essential for guiding the strategic execution of your data analytics project and should outline key milestones, delivery timelines, and the agile iterations needed to achieve your objectives. It serves as a blueprint for project execution, offering a clear path from inception to completion. ☐ Outline the major milestones, timelines, and agile iterations for your project. ☐ Create a communications plan for stakeholders along with a schedule for regular project update meetings. ☐ Identify risks that could delay or threaten delivery of the project, record in a risk register and develop mitigations to address those risks. ### 8\. Prepare for Change Management Your plan should aim to align the organization's culture and workflows with the new analytics capabilities, minimizing resistance and maximizing adoption. ☐ Anticipate how the introduction of new tools, processes, and data models will impact your organization. ☐ Develop a comprehensive change management plan to ensure a smooth transition. ☐ Ensure your team is fully onboarded and aligned with the project's vision and objectives. ## Do you need help with your data analytics project planning? Our Data Analytics Project Discovery Framework aims to provide long-term cost and time savings through an upfront investment in strategy and design, minimizing iteration cycles and delivering tangible business outcomes. It focuses on understanding and structuring strategy, conducting discovery to transform needs into empirical evidence, designing optimal information architectures and data models, and delivering a concise executive summary and agile roadmap for implementation. ![Rittman Analytics Discovery Plus Package Framework](https://rittmananalytics.com/lovable-uploads/471f53e8-59b2-4d64-89d5-51374318dec8.png) Intended to align the data strategy with business vision, it ensures designs and models are in place for execution, and setting a realistic and deliverable roadmap. [See client results ›](https://rittmananalytics.com/case-studies) 45 minutes · no obligation · you'll speak with a founder, not sales --- # RA Warehouse eCommerce Package for dbt - Pre-built Analytics Solution | Rittman Analytics Canonical: https://rittmananalytics.com/ra-warehouse-ecommerce URL: https://rittmananalytics.com/ra-warehouse-ecommerce Description: Stop guessing which marketing channels work. Our pre-built eCommerce analytics package for dbt Core delivers LTV, CAC, attribution, and cohort analysis in weeks, not months. Get unified data insights from Shopify, Google Analytics, and ad platforms. --- Start Driving Profitable Growth. Your e-commerce data is fragmented across Shopify, Google Analytics, and a dozen ad platforms. You're making critical budget and strategy decisions based on incomplete reports and manual spreadsheets, leaving revenue on the table. Our packaged eCommerce analytics solution takes you from fragmented data to a single source of truth for attribution, LTV, and cohort analysis in weeks, not months. [View Github Repo](https://github.com/rittmananalytics/ra_warehouse_ecommerce_v2) ## When Your Data Can't Keep Up with Your Customers As a modern e-commerce brand, you're interacting with customers across countless touchpoints. But without a unified data foundation, you're facing a set of familiar, high-stakes challenges. ### You're Flying Blind on Attribution You can't connect ad spend to the actual LTV of the customers they acquire. ### You Don't Know Your True Customer Value Calculating an accurate LTV or understanding your most valuable customer segments is a manual, error-prone process. ### Your Team is Drowning in Manual Reporting Your most valuable team members spend hours every week in spreadsheets, leaving no time for strategic analysis. ## Answer Your Most Critical E-commerce Questions, Out-of-the-Box The RA Warehouse for E-commerce is a customisable and extensible dbt package and set of example React dashboards that solves the core analytics challenges for D2C brands. It integrates your transactional, behavioral, and marketing data into a single, cohesive data model, ready for analysis. ### Customer Lifetime Value (LTV) Accurately calculated and sliced by acquisition cohort or channel ### Customer Acquisition Cost (CAC) A blended CAC across all channels for a true profitability picture ### LTV:CAC Ratio The ultimate measure of marketing efficiency ### Cohort Retention Analysis Track how well you retain customers from each monthly acquisition cohort over time ### RFM Segmentation Automatically segment your customers by Recency, Frequency, and Monetary value to identify your VIPs, at-risk customers, and more ![Executive Overview Dashboard showing key eCommerce metrics](https://rittmananalytics.com/lovable-uploads/dc3f27d9-3cf7-4209-bb4e-97e303a0e18b.png) ## Create a Single View of your eCommerce KPIs Everything you need to understand your customers, optimize marketing spend, and grow revenue, using our extensible and customisable dbt Package and LookML project ![eCommerce data flow diagram showing integration from platforms like Shopify, Google Ads, Facebook Ads to dbt and analytics outcomes](https://rittmananalytics.com/lovable-uploads/374f647b-8110-4e96-ad91-9ab1b2da1a5b.png) ## Accelerate Your Journey to Data-Driven Growth Building a custom data warehouse takes 6-12 months and is fraught with risk. Our pre-built solution delivers 80% of what you need in a fraction of the time, allowing you to focus on insights, not infrastructure. ### Rapid Time to Value Go from siloed data to a unified warehouse in 2-4 weeks. Our proven, pre-built data models are designed for rapid deployment and immediate business impact. ### Flexible & Extensible Our modular architecture allows you to use only the components you need and easily add custom metrics without breaking the core functionality. The open-source dbt code provides complete transparency and control. ### Best Practices Built-In Benefit from our years of expertise. We've built in best practices for identity stitching, incremental data processing, data quality testing, and comprehensive documentation, ensuring your platform is reliable and scalable from day one. ## Open-Source dbt Package for eCommerce Analytics The RA Ecommerce Data Warehouse v2 is a comprehensive dbt project that integrates data from multiple ecommerce and marketing platforms, following best practices for data modeling and providing extensive analytics capabilities. ### Key Features #### Multi-Source Integration Shopify, Google Analytics 4, Google/Facebook/Pinterest Ads, Klaviyo, Instagram Business #### Layered Architecture Staging → Integration → Warehouse with clear separation of concerns #### Data Quality Monitoring Comprehensive pipeline health tracking and data quality metrics #### Attribution Analysis Multi-touch customer journey attribution across all touchpoints #### Performance Analytics Campaign performance, customer segmentation, product analytics #### Comprehensive Testing 100+ data tests for data integrity and business rules Need help adopting, customizing, or extending the package to cover additional data sources? ### Open-Source & Extensible ![GitHub repository showing dbt package structure](https://rittmananalytics.com/lovable-uploads/67d05688-7803-4136-ba31-a62f1b5bd80a.png) ## Ready to Centralise Your eCommerce KPIs? Get fast time-to-value with a packaged analytics solution from Rittman Analytics [View Github Repo](https://github.com/rittmananalytics/ra_warehouse_ecommerce_v2) --- # A Preview of Fully-Agentic Project Delivery using Wire, Google Cloud and Rittman Analytics Canonical: https://rittmananalytics.com/blog/2026/10/17/a-preview-of-fully-agentic-project-delivery-using-wire-google-cloud-and-rittman-analytics URL: https://rittmananalytics.com/blog/2026/10/17/a-preview-of-fully-agentic-project-delivery-using-wire-google-cloud-and-rittman-analytics Description: At Rittman Analytics we’ve recently been experimenting with using Wire, our Claude Code plugin, to run teams of autonomous agents that work alongside our consultants when delivering analytics projects. Using… Author: Mark Rittman Published: 2026-10-17T22:29:58+01:00 Modified: 2026-10-17T22:33:22+01:00 --- ![A Preview of Fully-Agentic Project Delivery using Wire, Google Cloud and Rittman Analytics](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F10%2FScreenshot-2026-10-17-at-14.45.44.png) At Rittman Analytics we’ve recently been experimenting with using [Wire](https://docs.rittmananalytics.com/en/latest/), our Claude Code plugin, to run teams of autonomous agents that work alongside our consultants when delivering analytics projects. Using the same tools, project context and delivery methods as our consulting team, these agents have the potential to increase capacity while giving consultants more time for architecture, complex decisions and client relationships. It’s still early days and there are still kinks to work-through, but we have them working on internal projects today and to see how the future of analytics engineering might look in the near future, in the Slack screenshots below … I’m the only human in the conversation. Press enter or click to view image in full size ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fmiro.medium.com%2Fv2%2Fresize%253Afit%253A1400%2F1%252AHZzbtH252m6QQdnbh0HVQQ.png%3Fresize%3D700%252C504%26ssl%3D1) In this post I’ll talk about how we’re bringing this together, how these agents work under the covers and how we’re ensuring that using this type of AI technology is safe, controllable and adds value for our clients. ## What Are Wire Hosted Agents? [Wire Hosted Agents](https://docs.rittmananalytics.com/en/latest/docs/technical-reference/hosted-agents) are automomous, AI-powered agents that Rittman Analytics runs as members of its own delivery teams, and are only being deployed on internal project work for now. Each of these autonomous agents has its own Jira and Slack accounts and follows the same workflow, processes and steps that our human consulting team does: they read the engagement’s Wire record, propose a plan, write tickets according to best practices and ask when a decision is needed, then build the change with Wire’s commands and hand their work back as a pull request. Press enter or click to view image in full size ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fmiro.medium.com%2Fv2%2Fresize%253Afit%253A1400%2F1%252A0Tk5P2jtURXYC8x5-Yqv9w.png%3Fresize%3D700%252C637%26ssl%3D1) They differ from running Wire yourself in Claude Code in three ways: - **They are always on.** Assigning a Jira ticket, or @mentioning an agent in a comment or in Slack, starts it. Nobody needs a terminal open. - **They work in teams.** Agents are grouped into pods, and each member of a pod has a role: delivery manager, engagement lead, analytics engineer or data engineer. - **They work under a governance template.** Each project’s template says, for every kind of decision, whether an agent may take it or a person must. ### Hosted Securely on Google Cloud The infrastructure used to host these autonomous agents [runs on Google Cloud](https://docs.rittmananalytics.com/en/latest/docs/hosted-agents/platform-architecture) and provides a web-based admin console to create new agents, assign roles and personas and [group them into pods](https://docs.rittmananalytics.com/en/latest/docs/hosted-agents/setting-up-a-pod) for collaborative project work. Press enter or click to view image in full size ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fmiro.medium.com%2Fv2%2Fresize%253Afit%253A1400%2F1%252AlMqucB57aO4zb5WYECJl8w.png%3Fresize%3D700%252C546%26ssl%3D1) Each agent runs on its own Google Compute Engine machine so that they can have full access to a filesystem, terminal session, Claude Code and all of the tools including Wire that our regular consultants use on their projects. Each agent’s VM is built from a shared and version-controlled container image stored in Artifact Registry, and deployed onto a platform with its own private network and outbound communications run through a Cloud NAT. Credentials used by each agent are stored separately in Secret Manager with IAM controlling what each agent can access Press enter or click to view image in full size ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fmiro.medium.com%2Fv2%2Fresize%253Afit%253A1400%2F1%252Aw-eFEyt8MgJ0ky9ECt6TGg.png%3Fresize%3D700%252C376%26ssl%3D1) ### Agents and Humans working together in Jira and Slack A fundamental goal with our hosted agents platform is that humans interact with agents in the same way they interact with human users, and agents communicate between themselves (and leave project records) using the same tools, [agile project method](https://docs.rittmananalytics.com/en/latest/docs/hosted-agents/sprints-and-backlog) and Wire Plugin package that our human consultants use. Under the covers, assigning a Jira ticket or mentioning an agent in Slack then triggers a receiver on Cloud Run. It verifies that the event is genuine, and then then adds it to the agent’s Pub/Sub queue. Press enter or click to view image in full size ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fmiro.medium.com%2Fv2%2Fresize%253Afit%253A1400%2F1%252AXXYxeYjV98ZVxgNiP8b_pg.png%3Fresize%3D700%252C457%26ssl%3D1) A supervisor on the agent’s virtual machine then picks up the ticket and starts Claude Code with the Wire plugin, using Anthropic’s Claude models. Just as when you run the Wire in your own terminal session, the plugin then uses an [orchestration session to plan the work and delegate tasks to specialist subagents](https://docs.rittmananalytics.com/en/latest/docs/advanced/release-director) that in-turn use tools such as git, dbt and BigQuery to do their work. Agents, when they need to ask questions do so using Jira comments, Slack threads and replies and open pull requests on GitHub. A [governance template](https://docs.rittmananalytics.com/en/latest/docs/hosted-agents/governance) defines when an agent needs a person to make a decision. Administrators can monitor and manage agents through a console on Cloud Run, with shared state stored in Firestore and logs in Cloud Logging. ## An Example Fully-Agentic Project Delivery Starting delivery of an internal project using our Wire Hosted Agents platform is simply a case of @ mentioning an engagement lead agent in Slack and passing-over a statement of work detailing what needs to be delivered, and the degree of involvement that I (as the client) need to have in-terms of signs-offs and approvals. Press enter or click to view image in full size ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fmiro.medium.com%2Fv2%2Fresize%253Afit%253A1400%2F1%252A6utrWTbl0tcKAt8iD71D4w.png%3Fresize%3D700%252C397%26ssl%3D1) In this case I’ve chosen a [lightly-governed model](https://docs.rittmananalytics.com/en/latest/docs/hosted-agents/governance) for this project’s delivery, and so soon after the engagement lead agent instructs the delivery lead agent to break the work down-into tickets, select the appropriate Wire release type and then create and assign tickets in Jira, reporting-back at the end through Slack so that I’m kept in-the-loop as things progress. Press enter or click to view image in full size ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fmiro.medium.com%2Fv2%2Fresize%253Afit%253A1400%2F1%252AAOAYzKgGm-gaKDK3-U9BCA.png%3Fresize%3D700%252C538%26ssl%3D1) Agents then work on individual tickets that follow our Wire “release type” blueprints, asking supervisory agents to QA routine tasks and escalating more critical ones to human users, raise PRs and generally follow the software development lifecycle. Press enter or click to view image in full size ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fmiro.medium.com%2Fv2%2Fresize%253Afit%253A1400%2F1%252ACyBlJCvoHj8QsNgJzwBXnA.png%3Fresize%3D700%252C500%26ssl%3D1) To be clear and as I said earlier, this is an experimental feature that we’re still developing and figuring out how best to deploy on real client work, and will probably be something we end-up offering as a commercial service rather than bundling it into our free Claude Code plugin. Watch this space though as our thinking in this area evolves, and drop me an email at [mark.rittman@rittmananalytics.com](http://mark.rittman@rittmananalytics.com/) if your organisation might be interested in helping us as a design partner. ## Interested? Find Out More Rittman Analytics is a boutique data analytics consultancy that helps ambitious, digital-native businesses scale up their approach to data, migrate to Looker from legacy BI tools and accelerate their analytics projects using [Wire](https://rittmananalytics.com/services/wire-framework) and our other AI technologies. We’re authorised delivery partners for Google Cloud along with Oracle, Segment, Cube, Dagster, Preset, dbt Labs and Fivetran, and are experts at helping you design an analytics solution that’s right for your organisation’s needs, use cases and budget, and working with you and your data team to implement it. If you’re looking for help with data and AI needs, or would just like to talk shop and share ideas on what’s going on in your organisation and the wider data analytics world, [contact us now](https://calendly.com/markrittman/initial-discovery-call-with-mark-rittman) to organise a free, no-obligation call. We’d love to hear from you! --- # Migrating Tableau, Metabase and Data Studio Reports to Looker using AI, Wire & Rittman Analytics Canonical: https://rittmananalytics.com/blog/2026/10/11/migrating-tableau-metabase-and-data-studio-reports-to-looker-using-ai-wire-rittman-analytics URL: https://rittmananalytics.com/blog/2026/10/11/migrating-tableau-metabase-and-data-studio-reports-to-looker-using-ai-wire-rittman-analytics Description: One of the most-common requests we get from clients is to help them move from legacy BI tools such as Metabase, Tableau or Power BI to a modern data analytics… Author: Mark Rittman Published: 2026-10-12T00:13:42+01:00 Modified: 2026-10-12T00:13:42+01:00 --- ![Migrating Tableau, Metabase and Data Studio Reports to Looker using AI, Wire & Rittman Analytics](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F10%2F1b-bl4zdtAyHn0ArbdDr7A.png) One of the most-common requests we get from clients is to help them move from legacy BI tools such as Metabase, Tableau or Power BI to a modern data analytics platform and semantic layer using Looker. ![A series of analytical dashboards showcasing web analytics, report creation, and customer cohort performance using data visualisations and charts.](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F10%2F1b-bl4zdtAyHn0ArbdDr7A.png%3Fresize%3D1400%252C481%26ssl%3D1) Another, related type of request is to help a team move from a standalone reporting tool such as Google Data Studio (formerly, Looker Studio) to full Looker and its semantic layer, with the common goal being to provide better self-service analytics to users and do so with a single, integrated view of metric definitions and slice-and-dice dimensions. In many cases, moving to a new BI platform is part of a wider data transformation project where the tools for data extraction and transformation, along with the underlying data warehouse are all being upgraded and the BI tooling layer then becomes a re-implementation, often many months after the start of the project and with everything having to be re-built from scratch. But if the underlying data warehouse is fine right now and urgency is to get users off of those legacy BI tools and onto Looker before the next renewal date, refactoring those ‘like-for-like’ Looker reports later-on, our BI tool migration service enabled by AI automation can get this migration out-of-the way quickly and efficiently without tying-up all of your data team resources for months on-end. ## Using AI to help automate Legacy BI Tool Migration In [our blog last week](https://blog.rittmananalytics.com/whats-new-in-wire-4-x-multi-agent-development-dbt-charts-agents-schema-more-7d07acb467a7) I talked about the new release of [Wire](https://rittmananalytics.com/services/wire-framework), our AI delivery toolkit used by consultants at Rittman Analytics to safely accelerate the delivery of our client consulting projects using AI. By combining agentic coding tools such as Claude Code with our project delivery blueprints and “human-in-the-loop” judgement calls, our approach enables clients to reap the benefits of AI in their BI project delivery whilst retaining the benefits of our experienced, best-practices approaches. We also make Wire available as a [Claude Code plugin](https://github.com/rittmananalytics/wire-plugin) for non-competing use, and publish full documentation at [docs.rittmananalytics.com](https://docs.rittmananalytics.com/en/latest/docs/release-types/bi-migration) One of Wire’s capabilities is to assist us with BI tool migrations, and the 4.2.5 release of Wire introduces three new “migration pairs” for migrating two legacy and one standalone BI tool to Looker: 1. **Tableau to Looker** (4.2.4): Tableau Cloud, Server or Desktop workbooks become LookML views, explores and LookML dashboards. 2. **Metabase to Looker** (4.2.5): Metabase questions, models, metrics and dashboards become LookML views, explores, measures and LookML dashboards. 3. **Looker Studio to Looker** (4.2.5): Looker Studio (formerly Data Studio) reports become LookML, with each metric’s definition agreed first in a metric catalogue. Note that all three pairs are experimental in this release. Check out the [Release Notes](https://docs.rittmananalytics.com/en/latest/docs/reference/release-notes) for more details on this and other new features in Wire 4.x. ## How BI Tool Migration to Looker Works Though Tableau, Metabase, Data Studio (and in the near-future, Power BI and Oracle BI) all have their own way of defining and storing report definitions, most (though not all) provide APIs that expose those definitions and are well-suited to agentic processes that can introspect those definitions and then store them in a common format ready for conversion into a semantic layer and the equivalent reports in Looker. The diagram below shows the overall technical architecture our process follows, from the source tools on the left to Looker on the right and with a “parity loop” underneath. ![A flowchart illustrating a data migration process from various sources like Tableau, Metabase, and Looker Studio to Looker. The diagram includes steps for snapshot and audit, wire processing, conversion scripts, and deployment to Looker, with highlighted tools and components involved.](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F10%2F0jDnk3JrVxSQ0x29g.png%3Fresize%3D1400%252C746%26ssl%3D1) A tool-specific means of extraction is used for each of the legacy BI tool sources, via public APIs where available: - for **Tableau Cloud or Server**, we use their REST API with a personal access token. - for **Metabase**, again we use their API with an API key. - for **Google Data Studio**, which has no API for report definitions, we use Claude Code’s Chrome browser automation to introspect each report and its data source definitions, along with reading database query logs that Data Studio tags with the unique ID of each report. ### Using Agents to Automate the Migration Process The first job in any migration is getting a complete, pinned copy of what the source tool holds. Whatever the source, Wire’s audit runs the same six steps as shown in the diagram below. ![Diagram illustrating the steps of a data audit process, including stages such as Connect, Snapshot, Classify, Write report, Validate, and Review, along with their associated data outputs and file mentions.](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F10%2F1KfhsY1iJ6p5ocpT3Vx345w.png%3Fresize%3D1400%252C604%26ssl%3D1) The audit process then catalogues, classifies and flags for review everything the snapshot found, and passes it into Wire as a tool-neutral metric, report or dashboard definition. Once the audit is approved and the plan has your rulings, the converter part of the process turns each source item into its LookML equivalent. ![Diagram illustrating the flow from source items to LookML artifacts, highlighting processes such as joins, calculations, dashboards, and filters, concluding with output validation steps.](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F10%2F0aJ1engeMZzOinVhS.png%3Fresize%3D1400%252C700%26ssl%3D1) - **Tables or data sources** become `views/*.view.lkml` files, typed, with primary keys. - **Joins and relationships** become explore joins in `models/.model.lkml`. - **Calculations, metrics and aggregations** become measures and dimensions. Tableau LOD expressions and table calculations go to people. - **Native SQL, models and custom SQL** become derived-table views. Simple SQL is also lifted onto the table’s view, as a proposal. - **Dashboards, pages and tabs** become `dashboards/*.dashboard.lookml` files on the 24-column grid. - **Filters and controls** become dashboard filters with `listen`. ### Complementing Automation with Testing and Human Judgement **Human in the loop.** Every step also writes to a file that logs everything the converter cannot translate safely, rather than guessing and potentially changing a key number in a report. Each item goes through a plan ruling and then review so that Wire’s agents do as much mechanical translation as is safely possible, and it then works through the judgment calls with you. Finally, all output we send to Looker has to pass three layers of testing: 1. **Linting**, which for example checks for any source-tool syntax left in the LookML or a field defined twice; 2. **Looker’s validator**, checking the validity of Wire’s generated LookML; 3. **Parity**, which checks that every Looker dashboard tile produces the same results as its source before cutover. ## An Example BI Tool Migration using Wire So what does this look like in-practice? In the following example we’re working with a fictional retailer, Halcyon Outdoor, who has a set of Tableau, Metabase and Google Data Studio reports they want to bring-over to Looker as-is, allowing them to retire their legacy BI tools and refactor the resulting semantic layer at a later date. In this example: - Finance uses **Tableau Cloud**, and wants their revenue report moved over to Looker - the product team uses **Metabase**, and wants their web analytics dashboards and questions moved to Looker - and marketing reads **Data Studio** reports and wants those in Looker too ### Getting Started, and Prerequisites If you already have the Wire plugin installed, run `/reload-plugins` in Claude Code to pick up 4.2.5. If not, install it in three steps: /plugin marketplace add rittmananalytics/wire-plugin /plugin install wire@rittman-analytics /reload-plugins Each source needs one credential, kept in a file under `~/.wire/` outside your repository: - **Tableau**: a personal access token (your site administrator may need to allow token creation). - **Metabase**: an API key in a group that can see the content, with the tables synced. - **Looker Studio**: Chrome on your machine, and a record of who approved read-only access to the reports. ### Interacting with Wire in the 4.x Release Wire 4.x introduced a new “release director” way of working with Wire within Claude Code, where the user sets the session goal and makes key decisions along the way, rather than having to learn and directly type-in the commands that Wire uses to automate project delivery. We start by logging-into Claude Code and setting our overall project goals. ▐▛███▛█ Claude Code v2.1.289 ▝▜██████▀ Opus 5.5 (1M context) · Claude Team ▝▝ ▝▝ ~/github/halcyon-bi-to-looker❯ Halcyon is moving all its reporting to Looker. Three sources: Tableau Cloud (site halcyon, the Finance project), Metabase at halcyon.metabaseapp.com (the Product collection), and the three Looker Studio marketing reports in this list. Same BigQuery warehouse throughout; the LookML project is halcyon\_looker.⏺ Skill(wire:new) ⎿ Successfully loaded skill⏺ One release has one source tool, so I'll open three releases under this engagement, all writing to halcyon\_looker. Each pair is experimental. .wire/releases/02-tableau-to-looker/ bi\_pair: tableau\_to\_looker .wire/releases/03-metabase-to-looker/ bi\_pair: metabase\_to\_looker .wire/releases/04-looker-studio-to-looker/ bi\_pair: looker\_studio\_to\_looker halcyon\_looker already has 212 views, so each release writes to its own folder under its own model: tableau\_migration/, metabase\_migration/, looker\_studio\_migration/. Your existing models are not touched. One thing I need before the Looker Studio capture: who approved automated read-only access to the three reports, and when?❯ The CMO approved it this morning.⏺ Recorded in 04-looker-studio-to-looker/status.md. Ran: new (x3) This first step set-up the project scaffolding within the git repo, and once that setup is confirmed I then instruct Wire to start the audit process for all three BI tools. ❯ Run the audits.⏺ Tableau: signed in with the token, snapshot of the Finance project. 6 workbooks, 41 sheets, 9 dashboards, 11 data sources (4 published) data source fingerprints: 7 embedded copies collapse into 3 groups redesign: 2 LOD expressions, 1 table calculation, 1 extract-only data source file sources: 1 workbook reads an Excel file, not the warehouse Metabase: snapshot of the Product collection through the API. 23 questions (17 query builder, 6 native SQL), 2 models, 3 metrics, 4 dashboards (one with 3 tabs) every query-builder table synced; primary keys from metadata on 9 of 11 tables not converted: 2 questions binned by "number of bins", 1 multi-stage question Looker Studio: opening Chrome. Sign in when the window asks.❯ Done.⏺ Captured 3 reports (14 pages). The data sources are of three kinds: BigQuery tables 8 become LookML views BigQuery custom SQL 2 become views on their SQL Looker Studio's Looker connector 3 the charts already read Looker: they point at the existing explores Search Console, GA4 2 Looker cannot query them: listed for the plan Metric catalogue: 211 calculated fields in 64 groups; 9 metrics are defined two or more ways (conversion rate three ways). Ran: tableau-audit-generate, -validate; metabase-source-audit-generate, -validate; looker-studio-audit-generate, -validate; metric-catalogue-generate, -validate Once this initial audit has completed as much as it can on its own, it then stops as it needs the user to make some decisions where its next steps aren’t clear. ❯ What do you need from me?⏺ Eleven rulings across the three releases. The ones that change numbers first: R-1 Metabase "Weekly signups by channel" and 5 other native SQL questions: views on their SQL are exact but not reusable. I lifted the 2 simple ones onto the sessions view as a proposal. Remodel onto shared views, or keep? R-2 Tableau "Customer lifetime value" is a FIXED LOD expression. Derived table, or drop? R-3 Looker Studio "Conversion rate": three definitions (orders/sessions, orders/users, purchases/sessions). Which one is the company's? R-4 Looker Studio blend "Orders and sessions" (full outer join, no keys recorded): model the join on date and channel, or drop the 2 charts on it? R-5 Tableau "Regional targets" reads an Excel file: load it to the warehouse, or drop? ...❯ R-1 keep the 4 complex ones as views for now; take the lifted fields. R-2 derived table. R-3 orders over sessions. R-4 join on date and channel. R-5 load it as a dbt seed. Accept your proposals for the rest.⏺ Recorded R-1 to R-11 in the three plans and in the metric catalogue review. The plans are ready for your approval.❯ Approved. With the plans approved, the user tells Wire to build it. The three converters write LookML to a branch of the client’s LookML project repository, linting the output however before anything reaches Looker. \> Please build it⏺ Branch wire/bi-to-looker in halcyon\_looker. Converting: Tableau 9 views, 6 explores, 9 dashboards; 3 primary keys proposed from \*\_pk columns; the LOD as a derived table under hand/ (R-2) Metabase 11 views, 8 explores, 6 dashboards (one per tab); joins from Metabase's foreign keys; 3 metrics as measures with their filters; 4 views on native SQL (R-1) Looker Studio 8 views from BigQuery tables, 2 on custom SQL; "Conversion rate" as SAFE\_DIVIDE of orders over sessions (R-3); 14 dashboards. The 4 pages whose charts read the existing analytics model are in dashboards/analytics/: add one include line to that model to show them. Lint: 1 problem. metabase\_migration/views/sessions.view.lkml: field session\_start\_ts\_month is defined twice A field lifted from SQL took a name the session\_start\_ts date group already generates. Renamed it session\_start\_ts\_month\_from\_sql, re-ran: 0 problems. Pushed. Validate it in development mode, merge, and configure the three new models with halcyon\_bq. Ran: looker-target-setup-generate (x3), lookml-model-generate (x3), lookml-model-lint (x3), looker-content-generate (x3) The user validates the branch in Looker, merges and deploys it. Wire then compares every tile with its source. ❯ Deployed, models configured. Please test for equivalency. ⏺ Each source's rows for each tile, against the Looker element's rows, through the same comparator: Source Tiles PASS accepted difference FAIL Tableau (Query View Data) 41 40 0 1 Metabase (question data) 31 31 0 0 Looker Studio (recordings) 58 55 3 0 Tableau FAIL: "Margin by region" differs on every row. The Tableau calculation was a sum of row-level ratios (Tableau sums a row-level field by default); the converter emitted it that way and listed it. Sum of ratios or ratio of sums? Looker Studio accepted differences, for your approval: 2 tiles the chart's range ends today; Looker's nearest filter also counts rows dated later this month (forward-dated invoices). Same values for every closed month. 1 tile a breakdown chart: Looker Studio drew its top 15 series, Looker draws all 47. The 15 it drew match Looker's rows exactly. ❯ Ratio of sums: that was always the intent. Accept the three differences. Changed "Margin by region" to a ratio of sums under hand/; re-ran it against Tableau with the old number shown beside it, which the finance team should see before cutover. Recorded the three accepted differences with you as approver. All 130 tiles satisfy the cutover gate. Ran: bi-equivalency-validate (x3) The “Margin by region” result is the kind of thing a migration should surface as the Tableau number was wrong, the Looker number is right, but the finance team needs to hear that before cutover rather than after. The full session, including cutover planning and drift checks during the parallel run, is in the [terminal session walkthrough](https://docs.rittmananalytics.com/en/latest/docs/tutorials/bi-to-looker-session) in the Wire docs. ## So What Does Our Automated Process Cover? This feature is new and experimental and automation doesn’t cover all scenarios, which of course is where an experienced partner such as ourselves becomes invaluable. ### Tableau to Looker Migration Tableau keeps the model and the content together in one XML document per workbook, plus any published data sources. There is no git repository to read, so Wire takes a snapshot through the REST API (or reads a folder of \`.twb\` and \`.twbx\` files for a Desktop-only estate) and pins each object by its last-updated time and a hash of the file. What the Tableau converter handles: - **Columns and row-level calculations** become dimensions, date dimension groups and measures. `IF ... ELSEIF`, `ZN()`, `DATETRUNC` and `DATEDIFF` translate to BigQuery or Snowflake SQL. - **Aggregate calculations** such as `SUM([profit]) / SUM([sales])` become `` `type: number` `` measures. - **Groups and bins** become `CASE`and `FLOOR` dimensions, including bins sized by a parameter. - **Duplicate data sources** are matched by fingerprint (connection, tables, joins and columns, not names), so seven embedded copies of the same orders data become one shared explore if you agree. - **Primary keys**. Tableau’s relationships record no keys, so Wire proposes each view’s `` `*_pk` `` column as its primary key. Looker then uses symmetric aggregates, and totals stay correct across joins. - **Dashboards** land on Looker’s 24-column grid, with quick filters as dashboard filters. A sheet that is on no dashboard becomes a one-tile dashboard that keeps its sort and filters. - **LOD expressions, table calculations, blends and extracts** are listed for a ruling, never guessed at. As an example, we tested this migration pair on our own Tableau Cloud trial site, with a published BigQuery data source over five tables and a “Revenue by Company” sheet (blank names excluded, totals of £50,000 or more, largest first). ![A data visualisation with two sections: the top section shows a horizontal bar chart representing the revenue by company, featuring various company names along the vertical axis and total GBP amounts on the horizontal axis, and the bottom section displays a table listing the same companies with their corresponding total GBP amounts, using colour coding to highlight different revenue brackets.](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F10%2F1UWYQQGjws7aB9fr67l3sow.png%3Fresize%3D1400%252C946%26ssl%3D1) Wire converted it into a `tableau_migration` folder and model alongside the existing views in our LookML project, deployed it, and queried the tile through the Looker API. Looker’s 34 rows matched Tableau’s, every company and value, in the same order. Looker’s SQL used the proposed `invoice_pk` key to aggregate correctly over the join. Walkthrough: [Tutorial: Tableau to Looker](https://docs.rittmananalytics.com/en/latest/docs/tutorials/tableau-to-looker). ## Metabase to Looker Metabase keeps typed metadata for every table it has synced including each field’s type and its primary and foreign keys. Questions built in the query builder refer to fields by id, so the converter knows which column each question reads, which LookML view it belongs to and how the tables join. What the Metabase converter handles: - **Tables** become views, typed and keyed from Metabase’s metadata. - **Query-builder questions** become shared explores, with joins taken from Metabase’s foreign keys rather than inferred. - **Metrics** become measures that keep their filters (“Revenue, excluding cancelled orders”). - **Native SQL questions and models** become views on their SQL, with template tags as parameters. Simple single-table SQL is also lifted onto the table’s view, as a proposal for you to accept or reject. - **Relative dates** translate both ways: “previous 12 months” and “the last 12 months including this one” are different filters in Looker, and Wire writes the right one. - **Dashboards** map one-to-one onto Looker’s 24-column layout. Each tab becomes its own LookML dashboard, and dashboard filters keep their field mappings. We tested this converter on a Metabase Cloud instance with a BigQuery database: a “Web Analytics” dashboard with two native SQL questions (sessions by month, bounced session rate by channel), and a second dashboard of two questions built in the query builder, one with a Channel filter. All four tiles matched in Looker, row for row and in the same order. ![Top section displaying web analytics metrics with graphs for bounced session rate by channel and sessions by month.](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F10%2F1AobXspE1gNYCvdMN57a4Bg.png%3Fresize%3D1400%252C1262%26ssl%3D1) Looker’s validator also caught a fault our test fixtures had missed: a field lifted from SQL took a name that a date dimension group already generates. The converter now avoids every generated name, and Wire’s lint step fails on any field defined twice, so the problem is caught before the LookML reaches Looker. Walkthrough: [Tutorial: Metabase to Looker](https://docs.rittmananalytics.com/en/latest/docs/tutorials/metabase-to-looker). ### Google Data Studio to Looker Looker Studio is the hardest of the three, because it has no shared model at all. Each report has its own data sources and its own calculated fields, so “conversion rate” is often defined three different ways across three reports. Converting them one for one would carry those differences into Looker. So this pair starts with a capture and a metric catalogue. Wire opens each report in the editor and records its definitions, and the values each chart showed. It then groups every calculated field by metric and classes each group as a copy, an alias or a conflict. You (or better, the client’s metric owners) rule on one definition per metric, and the converter writes LookML from those rulings. In this conversion: - **A BigQuery table** becomes a LookML view, with measures for the charts’ aggregations and the agreed calculated fields. - **A BigQuery custom SQL data source** becomes a view on its SQL. - **Looker Studio’s Looker connector** needs no new LookML: the charts already read Looker, so they point at the existing explore. - **Search Console, GA4, Sheets and other connectors** are listed for the plan: load the data to the warehouse first, or drop. Calculated fields become BigQuery SQL, with Looker Studio’s division written as `SAFE_DIVIDE`, because Looker Studio returns null when dividing by zero and BigQuery raises an error. Each report page becomes a LookML dashboard, with the date range and filter controls as dashboard filters. Parity compares each chart’s recorded values with the Looker element’s rows, which works for every connector. We tested this converter it on two of our own reports, one of which reads our existing Looker model through Looker Studio’s Looker connector and the other that read two BigQuery tables and a custom SQL query. For the first report, six of its seven charts matched exactly. On the seventh, the chart’s date range ends today and Looker’s nearest filter also counts invoices dated later this month, so every closed month matched and only the current month differed. A breakdown chart showed another known difference in that Looker Studio drew only its top 15 companies, while Looker draws every row and those 15 matched exactly. For the second report, all four converted charts matched. The fifth chart, built on a blend, was listed for a ruling rather than converted. Walkthrough: [Tutorial: Looker Studio to Looker](https://docs.rittmananalytics.com/en/latest/docs/tutorials/looker-studio-to-looker). ## Interested? Find Out More Rittman Analytics is a boutique data analytics consultancy that helps ambitious, digital-native businesses scale up their approach to data, migrate to Looker from legacy BI tools and accelerate their analytics projects using [Wire](https://rittmananalytics.com/services/wire-framework) and our other AI technologies. We’re authorised delivery partners for Google Cloud along with Oracle, Segment, Cube, Dagster, Preset, dbt Labs and Fivetran, and are experts at helping you design an analytics solution that’s right for your organisation’s needs, use cases and budget, and working with you and your data team to implement it. If you’re looking for help with your migration to Looker, or would just like to talk shop and share ideas on what’s going on in your organisation and the wider data analytics world, [contact us now](https://calendly.com/markrittman/initial-discovery-call-with-mark-rittman) to organise a free, no-obligation call. We’d love to hear from you! --- # What’s New in Wire 4.x : Multi-Agent Development, dbt Charts, Agents Schema & More Canonical: https://rittmananalytics.com/blog/2026/10/11/whats-new-in-wire-4-x-multi-agent-development-dbt-charts-agents-schema-more URL: https://rittmananalytics.com/blog/2026/10/11/whats-new-in-wire-4-x-multi-agent-development-dbt-charts-agents-schema-more Description: We just released the 4.1.0 version of Wire, the plugin for Claude Code and Gemini CLI that brings our delivery blueprints, skills and best-practices to the AI-powered agentic coding tools… Author: Mark Rittman Published: 2026-10-12T00:08:10+01:00 Modified: 2026-10-12T00:08:10+01:00 --- ![What’s New in Wire 4.x : Multi-Agent Development, dbt Charts, Agents Schema & More](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F10%2F1ll3tZHaKFmujJAuTWqlxRQ.png) We just released the 4.1.0 version of [Wire](https://rittmananalytics.com/services/wire-framework), the plugin for Claude Code and Gemini CLI that brings our delivery blueprints, skills and best-practices to the AI-powered agentic coding tools provided by our partners at Anthropic and Google. Our consulting team uses Wire to deliver client projects that [build and upgrade your data platform](https://rittmananalytics.com/services/data-platform-development), [migrate your data warehouse](https://rittmananalytics.com/services/data-warehouse-migration) and [provide your AI agents with trusted numbers, evals and context layers](https://rittmananalytics.com/blog/2026/06/11/making-agentic-analytics-more-accurate-using-anthropics-agentic-data-stack-and-the-wire-framework). And as with previous releases of the [Wire Plugin](https://github.com/rittmananalytics/wire-plugin), we also make it available as a public resource to our clients, individual developers and internal data teams to check-out and ([non-competing](https://github.com/rittmananalytics/wire-plugin?tab=License-1-ov-file)) use, with full details including installation steps and worked examples available at [docs.rittmananalytics.com](https://docs.rittmananalytics.com/). ![Website interface showcasing the Wire Framework, highlighting AI-accelerated data platform engagements with command examples and features.](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F10%2F17HJyltxzyC6C3JJvARhZ1Q.png%3Fresize%3D1400%252C926%26ssl%3D1) ## What’s New in Wire 4.x Wire 4.x introduces a large number of new user-facing and back-end features, including: - [Business rules discovery](https://docs.rittmananalytics.com/en/latest/docs/advanced/business-rules), an optional first step in several release types that records, per business domain, every competing definition of a metric together with the file it came from, what they disagree on, the decision and who approved it. - [dbt Charts](https://docs.rittmananalytics.com/en/latest/docs/advanced/dbt-charts), a dashboard layer written as YAML files that live in the dbt project next to the models and can be run locally, or uploaded and hosted in dbt Labs’s new [dbt Charts managed service](https://dbtcharts.com/accounts/signup/) ![A dashboard displaying marketing analytics, including web conversions by month and channel, email statistics, LinkedIn post engagement, and campaign performance metrics.](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F10%2F1dc0TurHwajL5ClEcfhomJw.png%3Fresize%3D1400%252C1424%26ssl%3D1) - [Agents Schema](https://github.com/dbt-labs/agents_schema), dbt Labs’ new standard for putting the context an AI agent needs to query a warehouse **inside that warehouse**, in a schema named `AGENTS` - Check-out the [Release Notes](https://docs.rittmananalytics.com/en/latest/docs/reference/release-notes) in the Wire documentation for everything-else that’s new this latest release. ## Release Director Model : A New Multi-Agent Architecture in Wire 4.x The headline feature that underpins much of what’s new in Wire 4.x is a new multi-agent architecture that makes Wire both simpler to use and significantly more capable. We call this the [Release Director Model](https://docs.rittmananalytics.com/en/latest/docs/advanced/release-director). In the previous releases of Wire, the way that users delivered, for example, a project that led with a dashboard mock which was then used as a basis for gathering detailed requirements and turning them into a full working Looker dashboard, was to use as series of Wire commands in Claude Code like this: \# Phase 1: Requirements (Day 1) /wire:business-rules-generate /wire:business-rules-validate /wire:business-rules-review /wire:requirements-generate /wire:requirements-validate /wire:requirements-review \# Phase 2: Interactive Dashboard Mocks (Day 1–2) /wire:mockups-generate /wire:mockups-review \# Phase 3: Visualization Catalog (Day 2) /wire:viz\_catalog-generate \# Phase 4: Data Model (Day 2–3) /wire:data\_model-generate /wire:data\_model-validate /wire:data\_model-review \# Phase 5: Seed Data (Day 3) /wire:seed\_data-generate /wire:seed\_data-validate /wire:seed\_data-review \# Phase 6: Development — seed-based (Days 3–5) /wire:dbt-generate /wire:dbt-validate /wire:utils-run-dbt /wire:dbt-review Over time, though, we found that users preferred to tell Wire what they wanted in plain conversational language rather than memorise dozens of commands, options and modifiers across an ever-growing range of project types. Around the same time, our [agentic data migration work with Carwow’s data team](https://rittmananalytics.com/case-studies/carwow) pushed us in the same direction. Migrating their Snowflake data warehouse to Google BigQuery meant coordinating dozens of Claude agents working in parallel to translate dbt models and other components of the platform. That led us to a new way of working with Wire. Instead of asking the user to choose and run individual commands, they worked through a single orchestration session in Claude Code. That session interpreted what the user wanted, broke the work into smaller tasks and delegated those tasks to specialist agents, which then ran the appropriate Wire commands on the user’s behalf. ![Flowchart illustrating the roles and processes in a project release management system, highlighting the Release Director, Orchestrating Session, and various developer lanes including dbt developer, QA agent, semantic layer developer, and migration specialist.](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F10%2F103NKkOlbkLGVCikdreqXOw.png%3Fresize%3D1400%252C892%26ssl%3D1) This more hands-off and directional model of working with Wire was so successful and more easily adopted by users, that we’ve now made it the default for all types of work in this new 4.x release. ### Multi-Agent Development in Wire, In-Practice So what does this new, agentic way of working with Wire look like? Let’s return to the same scenario: the user wants to start with a quick dashboard mock based on an initial set of client requirements, use that mock to refine and confirm those requirements, then progressively build it out into a working prototype that can ultimately be refactored into a production Looker dashboard. ▐▛███▛█ Claude Code v2.1.289 ▝▜██████▀ Opus 5.5 (1M context) · Claude Team ▝▝ ▝▝ ~/github/wire-4-dashboard-first-demo Get to finished work sooner with Opus 5.5. Switch anytime with /model. ❯ i would like to create a looker and bigquery platform for core dynamics, but start by prototyping a marketing analytics dashboard starting with a mock, develop that rapidly and then get onto a wider program of work once that prototype (PoC) gets delivered and we win hearts-and-minds at the company. Can you start this engagement with this dashboard first approach, focusing on marketing analytics and whatever requirements you can find for this topic area in the SoW I've provided ... ⏺ The SoW is in the repo root. I'll start the engagement with the Wire new command, which sets up the engagement and the first release. ⏺ Skill(wire:new) ⎿ Successfully loaded skill · claude-fable-5 ... Read 1 file (ctrl+o to expand) ⏺ I've read the SoW. Most settings come straight from it. Four choices are yours to make before I create anything. Wire reads through the SoW (Statement of Work) and works-out what questions the user needs to answer before it can proceed further: ← ☐ Build route ☐ Tracker ☐ Doc store ☐ Client domain ✔ Submit → How should the marketing PoC be built? ❯ 1. Seeded (Recommended) Build the mock, data model, Looker views and dashboards on generated sample data first. Repoint to real HubSpot, Salesforce and ad-platform data later. Fits a PoC: no wait for Fivetran or GCP access. 2. Live data Skip sample data and build straight against real source tables. Only works if marketing data is already landing in BigQuery. 3. Type something. ──────────────────────────────────────────────────────────────────────────────────────────────────────────── 4. Chat about this Enter to select · Tab/Arrow keys to navigate · Esc to cancel The user answers this and the other outstanding questions and tells Wire to go-ahead, and after a while Wire confirms that it has setup the scaffolding for this project release and confirms what it knows and what information is still outstanding from its review of the documentation so far. ⏺ I've set up the Core Dynamics engagement in Wire, with the marketing analytics proof of concept (PoC) as its first release. The release is "dashboard-first": mock the dashboards, agree them with marketing, build on sample data, then switch to real data. Nothing has been built yet; the next step is turning the SoW's marketing section into a requirements document. ⏺ There is no issue tracker, no document store and no automatic call-transcript sync, as you chose. Marketing requirements in the SoW (section 5.2) - Problem: about $2.1M a year in marketing spend, with only last-touch attribution in HubSpot. HubSpot and Salesforce are linked only by a weekly manual CSV export. - 3 dashboards: - MA-01 Demand Generation Performance (weekly) - MA-02 Multi-Touch Attribution (5 attribution models to choose from) - MA-03 Pipeline Contribution (monthly, for executives) - 6 business questions, covering: - lead-to-sales conversion by channel - customer acquisition cost (CAC) by channel - pipeline generated per dollar of spend - the full funnel by cohort - how long deals sit at each stage - which content appears in won deals - 6 warehouse tables: funnel events, campaign spend, attribution touches, revenue split across touches, campaign, and contact. - 7 data sources: - HubSpot - Salesforce - Google Ads, LinkedIn Ads and Meta Ads - Outreach.io - Clearbit Open items, recorded in the context file 1. Transformation tool: the SoW names Dataform, but Wire's build step produces dbt. This needs a decision before the build starts. 2. Timing: the SoW schedules marketing dashboards for weeks 15 to 16, after the data foundation is built. Bringing them forward should be agreed with Priya Nair and Rachel Summers, because payment milestone M4 depends on them. 3. Salesforce to HubSpot sync: SoW assumption 9 says attribution depends on this sync keeping contact IDs consistent. It has not been checked yet. Next step: run /wire:requirements-generate 01-marketing-analytics-poc. Once Rachel Summers approves the requirements, the dashboard mockups come next. Should I start the requirements now and commit the setup? The user then checks what’s been produced, and then tells Wire to “drive for it” going forward, the cue to have the orchestration agent start using its fleet of specialist sub-agents to handle delivery for the user. ❯ yes please drive for me, go ahead ⏺ Skill(wire:release-director) ⎿ Successfully loaded skill ⏺ Got it — I'll proceed with committing the setup and drafting the requirements, then pause for approval before moving further. Logging this plan and the release claim now. ⏺ Setup committed. Now drafting the requirements from the SoW's marketing section. ⏺ Skill(wire:requirements-generate) ⎿ Successfully loaded skill ⏺ I'll delegate the drafting to the discovery analyst agent, passing along your stated priorities (marketing first, then the wider programme) rather than asking again. ⏺ Dispatching the requirements lane now. Then, going-forward, Wire’s orchestrating agent session will start handing-off tasks to [specialised Wire agents](https://docs.rittmananalytics.com/en/latest/docs/advanced/wire-agent-architecture) working within their own lanes, each of them under the control of the sole orchestration agent. Those lane agents will then run the Wire commands that perform the steps in this release type, make use of Wire Skills and in some cases use tools exposed by the various MCP servers connected-into your Claude Code environment. And, as you can see from the more detailed diagram below and [in our documentation](https://docs.rittmananalytics.com/en/latest/docs/advanced/wire-agent-architecture), components within each layer in the stack also write to logs, update status files and maintain state files so that the whole process is auditable and recoverable. ![Flowchart detailing a structured release management process including roles, commands, and documentation within a software development environment.](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F10%2F1ENkggxt5kuHGN6rUzau0bg.png%3Fresize%3D1400%252C1640%26ssl%3D1) Going back to the terminal, Wire’s orchestration session starts by spawning a discovery-analyst agent to go-off and look further-into the requirements we’ve been given by the client. ⏺ wire:discovery-analyst:AGENT(Draft and validate marketing requirements) ⎿ Backgrounded agent (↓ to manage · ctrl+o to expand) ⏺ I've committed the setup and started the requirements draft. It is running in the background and has not finished yet. The draft works from the marketing section of the SoW. It uses the priorities you set: ┌───────────┬──────────────────────────────────────────────────────────────────────────────────────────┐ │ Priority │ What it covers │ ├───────────┼──────────────────────────────────────────────────────────────────────────────────────────┤ │ Primary │ Marketing proof of concept: the 3 dashboards (MA-01 to MA-03), the 6 marketing questions │ │ │ and the 6 marketing tables │ ├───────────┼──────────────────────────────────────────────────────────────────────────────────────────┤ │ Secondary │ Only the platform and Looker pieces the proof of concept needs │ ├───────────┼──────────────────────────────────────────────────────────────────────────────────────────┤ │ Future │ Product, Customer, Operational and governance work, each listed as a later release │ └───────────┴──────────────────────────────────────────────────────────────────────────────────────────┘ The draft will also list the 3 open items as questions for the client, plus any unclear SoW terms, such as how MQL and SQL are defined. When it finishes, I'll check the file and update the status record. I'll then bring you the summary for approval. You can approve it now, ask for changes, or hold it for Rachel Summers to sign off. Running: /wire:requirements-generate 01-marketing-analytics-poc, /wire:requirements-validate 01-marketing-analytics-poc ### New in Wire 4.1.0 : Wire Studio Keeping-track of what’s Wire’s doing for you now the work is being handed-off to agents in their own session lanes, and so the 4.1.0 release introduces a new web-based viewer for what’s going on in your Wire session called Wire Studio. In the screenshot below we can see the the discovery agent running, gathering requirements that will once complete, be passed-back to the orchestration agent and made available for you to review in the project filesystem. ![Screenshot of a web interface displaying the status and steps of a project in a decision-making tool, including tasks and notes for the 'discovery-analyst' lane.](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F10%2F1MTompTu3egkX44ir9zauiA.png%3Fresize%3D1400%252C987%26ssl%3D1) After a few minutes, the Wire’s orchestration session reports back on what’s been determined, what open questions remain and asks whether you’d like to park those for a moment and move on to the next two stages in release workflow. ⏺ Agent "Draft and validate marketing requirements" finished · 6m 48s ⏺ The work checks out — 19 functional requirements, 6 non-functional requirements, and 3 deliverables are on disk, with only 2 minor deviations noted. I'll update the record now. ⏺ The requirements draft is done and passes all 11 checks. It needs your approval before the dashboard mockups can start. What the draft contains (.wire/releases/01-marketing-analytics-poc/requirements/requirements\_specification.md, committed) ┌──────────────────────────────────────────────────────────────────────────────┬───────┐ │ Item │ Count │ ├──────────────────────────────────────────────────────────────────────────────┼───────┤ │ Functional requirements: what the dashboards and metrics must do │ 19 │ ├──────────────────────────────────────────────────────────────────────────────┼───────┤ │ Non-functional requirements: speed, access, refresh times, metric governance │ 6 │ ├──────────────────────────────────────────────────────────────────────────────┼───────┤ │ Deliverables in this release (dashboards MA-01, MA-02, MA-03) │ 3 │ ├──────────────────────────────────────────────────────────────────────────────┼───────┤ │ Open questions for Core Dynamics │ 12 │ └──────────────────────────────────────────────────────────────────────────────┴───────┘ Product, Customer and Operational analytics are listed as deferred, each with a suggested later release. Open questions that change the numbers. The ones marked † are needed before the mockups can show realistic figures: - † How Core Dynamics defines an MQL and an SQL (Rachel Summers, Niamh Collins) - † The rule for "marketing-sourced" versus "sales-sourced" pipeline (Rachel Summers, David Park) - Customer acquisition cost (CAC) is defined as sales plus marketing spend. Sales cost is not in any of the 7 marketing sources (Rachel Summers, James Petit). - What counts as a "new logo" for CAC, and whether CAC uses a trailing 90 days or quarters - How far back a touch counts towards attribution, and which content assets are tracked - The 3 items we already knew: Dataform or dbt, the timing change, and the Salesforce to HubSpot ID sync 2 small problems I found while checking the lane's work - The document has 3 empty "reserved" sections (11 to 13), and the later sections are numbered 14 to 17 instead of 11 to 13. This is a layout fault only. Fixing it means re-running the draft. - The lane added notes to decisions.md, which was outside the files it was allowed to write. The notes are accurate, and I've recorded the slip in the run log. My recommendation: approve it internally now. In a dashboard-first release, the mockup sessions with Rachel Summers are the natural place to settle the MQL, SQL and sourcing questions. Holding the draft for her formal sign-off first would delay the mock that is meant to win support. The questions stay listed as open either way. The choice is yours: 1. Approve now. I record the approval under your name and start the mockups with you. 2. Changes. Tell me what to change, for example the section numbering, and I'll re-run the draft. 3. Hold for Rachel Summers to sign off. I record it as waiting on her, and nothing moves until she approves. Ran: /wire:requirements-generate 01-marketing-analytics-poc, /wire:requirements-validate 01-marketing-analytics-poc Not run: /wire:mockups-generate 01-marketing-analytics-poc, gate: requirements review ❯ Approve now Once approval is given, you can see in Wire Studio where we are in the overall workflow ![Overview of the release process in Wire Studio, displaying completed and runnable artifacts, including business rules, requirements, and design elements such as conceptual models and mockups.](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F10%2F1T-lpMBwDwbJVolzm4OXQew.png%3Fresize%3D1400%252C987%26ssl%3D1) and the orchestration agent now creates two more agents, one to create the conceptual data model that such a dashboard would require, and another to design and build the actual dashboard mock. Requirements approved and committed. The design phase is now open. Here is what can run next. Run plan: 01-marketing-analytics-poc, up to the mockup and conceptual model reviews #: 1 Step: Build the dashboard mockups with you Type: command (foreground, interactive) Command: /wire:mockups-generate 01-marketing-analytics-poc Scope: design/mockups/ Produces: Clickable Looker-style HTML mocks of MA-01, MA-02, MA-03, refined with you ──────────────────────────────────────── #: 2 Step: Draft the conceptual model (the business entities and how they relate: campaign, contact, touch, lead, opportunity, spend) Type: lane (data-designer) Command: /wire:conceptual\_model-generate 01-marketing-analytics-poc Scope: design/conceptual\_model\* , lanes/conceptual\_model.state.md Produces: Entity model and its check report ──────────────────────────────────────── #: 3 Step: Approve mockups and conceptual model Type: decision Command: director Scope: mockups review, conceptual\_model review Produces: Your rulings; the run stops here Steps 1 and 2 run in parallel, because neither depends on the other. Not planned: - /wire:viz\_catalog-generate 01-marketing-analytics-poc and /wire:data\_model-generate 01-marketing-analytics-poc. Both need the mockups approved first. - /wire:business-rules-generate 01-marketing-analytics-poc. This is optional (see the question below). One ruling is needed before step 2. The conceptual model has an advisory check: it expects agreed business rules first. Those rules would cover how MQL, SQL, marketing-sourced pipeline and CAC are defined, which is open questions C-3 to C-7. There are 2 options: - Skip for now (my recommendation). Draft the model with those definitions marked as placeholders, and settle them with Rachel Summers in the mockup sessions. I record the skip and the reason as ruling R-2. - Run business rules first. This adds /wire:business-rules-generate before step 2. It is the better route if you want the definitions agreed formally before any design. It needs time from Rachel Summers and Niamh Collins, which slows the proof of concept. go (with skip) / go (with business rules first) / adjust / cancel? Not run: /wire:mockups-generate 01-marketing-analytics-poc, /wire:conceptual\_model-generate 01-marketing-analytics-poc, gate: waiting for your go on the plan ⏺ wire:data-designer:AGENT(Draft and validate conceptual model) ⎿ Backgrounded agent (↓ to manage · ctrl+o to expand) ⏺ The conceptual model lane is running in the background. Starting the mockups with you now. Going back to Wire Studio, you can see both of these agents, running in their own Claude Code lanes, running Wire commands and delivering the tasks for you. ![Screenshot of a software interface displaying the 'Lanes' section of a project dashboard, showing various lane states including 'conceptual\_model.state', 'mockups.state', and 'requirements.state' with their respective statuses and last write timestamps.](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F10%2F1ll3tZHaKFmujJAuTWqlxRQ.png%3Fresize%3D1400%252C987%26ssl%3D1) After a while, the data-designer lane agent completes it’s work. ✻ Waiting for 2 background agents to finish ⏺ The conceptual model lane has finished. Checking its work before updating the record. ⏺ The conceptual model is done: 13 business entities and 18 links between them, passing all 12 checks. It is committed in design/conceptual\_model.md. The mockup agent is still running. What the model covers - Channels, campaigns and daily spend. - Contacts, the companies they work for (with Clearbit firmographic data) and opportunities. - Every marketing touch: ads, forms, emails, page views, content and SDR outreach. - Funnel stage changes, and an attribution table that splits each opportunity's value across touches under all 5 models. - It covers all 6 marketing tables named in the SoW. Design choices the agent made: - A "lead" is a contact at an early funnel stage, not a separate entity. - Outreach.io SDR touches are folded into the single marketing touch record. The 9 definitions you chose to settle in the mockup sessions (ruling R-2) are marked as placeholders. The Salesforce to HubSpot contact link is flagged as the main risk. 2 points to raise at review, not errors: - The text says SDR touches roll up into marketing touches, but the diagram does not draw that link. - No direct link between a contact and an opportunity. Attribution has to reach an opportunity through the company the contact works for, which is less precise than Salesforce contact roles. This is worth a decision before the data model. I'll hold this review until the mockups arrive, so you can review both together. The mockups may change the model. Ran: /wire:conceptual\_model-generate 01-marketing-analytics-poc, /wire:conceptual\_model-validate 01-marketing-analytics-poc Running: /wire:mockups-generate 01-marketing-analytics-poc Looking at the resulting conceptual model in Github, you can see how Wire has designed our conceptual model for us based on everything that’s been gathered, confirmed and recorded as required for the marketing analytics dashboards this release is designed to deliver. ![Entity Relationship Diagram illustrating relationships between various entities including Channel, Campaign, Contact, and Account with labelled connections.](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F10%2F119YQNF4SIwhTVKdoWt3oGA.png%3Fresize%3D1400%252C987%26ssl%3D1) A few minutes later, Wire’s orchestration session tells-us that the dashboard mocks we’re going to show to our stakeholder are now ready for viewing. The three dashboard mockups are built and open in your browser. My checks found no problems, apart from minor dashes in some chart titles. They're committed to design/mockups/, together with a list of all 27 charts, tiles and tables and a dashboard spec. What each dashboard shows (sample data sized to Core Dynamics: about $175K a month of marketing spend) ┌────────────────────┬─────────────────────────────────────────────────────────────┬──────────────────────────────────────────────────────────────────────────────────┐ │ Dashboard │ Headline tiles │ Charts and tables │ ├────────────────────┼─────────────────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────────────────────────┤ │ MA-01 Demand │ Leads, MQLs, MQL to SQL rate, cost per MQL (about $392), │ Funnel volume by week; conversion by stage; spend this month vs last; cost per │ │ Generation │ pipeline generated │ MQL and SQL by channel, with trend lines; top 10 campaigns │ ├────────────────────┼─────────────────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────────────────────────┤ │ MA-02 Attribution │ Attributed revenue, active model, touches to close, days to │ Revenue by channel, which changes when you pick an attribution model (try the 5 │ │ │ close, content-influenced wins │ buttons); days per stage by channel; touches by segment; content in won deals │ ├────────────────────┼─────────────────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────────────────────────┤ │ MA-03 Pipeline │ Marketing-sourced pipeline, influenced revenue, CAC (about │ Marketing-sourced vs sales-sourced by month; CAC by quarter against the 15% │ │ Contribution │ $31K), return on spend (about 7.3 to 1), new logos │ reduction target; return on spend by quarter │ └────────────────────┴─────────────────────────────────────────────────────────────┴──────────────────────────────────────────────────────────────────────────────────┘ Each figure that depends on an unagreed definition has a small amber "Def to confirm" note. These cover MQL and SQL, the sourcing rule, CAC inputs and period, what counts as a new logo, the lookback window, and content assets. Things to know: - Your commit. At 17:40 you committed and pushed 2 of the mockups while the agent was still writing them. My later commits sit on top of yours, and I have not pushed them. Should I push? - Dashes. The mockups use dashes in some chart titles and campaign names, against the house style. I can have them removed in the next round. Next, the review. The mockups are the point of this release, so this is where we work through them together. For each dashboard you can: 1. Approve as is. Once all 3 are approved, the chart list and data model can start. 2. Tell me what to change. For example a tile, a chart type, a filter, or how the numbers look. I'll send the changes back to the mockup agent and reopen the updated files. 3. Hold for Rachel Summers. I record the review as waiting on her sign-off. The conceptual model can wait until the mockups settle, since mockup changes may alter it. What would you like to change, if anything? Ran: /wire:mockups-generate 01-marketing-analytics-poc Not run: /wire:viz\_catalog-generate 01-marketing-analytics-poc, gate: mockups review ### Automatically-Generated Looker Dashboard Mocks Google Chrome has automatically opened and is now showing, on three separate tabs, fully-clickable and realistic Looker dashboard mocks (with open items clearly labelled) ready for running past our Stakeholder, showing value in hours rather than the days or weeks this type of exercise would have taken in the past. ![Dashboard showing marketing and sales pipeline contribution report with financial metrics including revenue, marketing ROI ratio, and CAC trends over several months.](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F10%2F1U6ErH7FJzQZ_XcnW_Wz0Aw.png%3Fresize%3D1400%252C987%26ssl%3D1) ## Interested? Find Out More [Rittman Analytics](https://rittmananalytics.com/) is a boutique data analytics consultancy that helps ambitious, digital-native businesses scale-up their approach to data, analytics and generative AI using [Wire](https://rittmananalytics.com/services/wire-framework) and our other AI technologies. We’re authorised delivery partners for Google Cloud along with Oracle, Segment, Cube, Dagster, Preset, dbt Labs and Fivetran and are experts at helping you design an analytics solution that’s right for your organisation’s needs, use-cases and budget and working with you and your data team to successfully implement it. If you’re looking for some help and assistance with your AI initiative or would just like to talk shop and share ideas and thoughts on what’s going on in your organisation and the wider data analytics world, [contact us now](mailto:hello@rittmananalytics.com) to organise a 100%-free, no-obligation call. We’d love to hear from you! --- # Agentic Data Platform Migration using Wire, Claude Code and Rittman Analytics Canonical: https://rittmananalytics.com/blog/2026/07/19/agentic-data-platform-migrations-using-wire-claude-code-and-rittman-analytics URL: https://rittmananalytics.com/blog/2026/07/19/agentic-data-platform-migrations-using-wire-claude-code-and-rittman-analytics Description: Migrating an entire analytics platform including the data warehouse, data source extractions, semantic layer, BI reports, orchestration and reverse ETL syncs from one stack to another used to be the… Author: Mark Rittman Published: 2026-07-20T00:52:42+01:00 Modified: 2026-08-22T16:48:35+01:00 --- ![Agentic Data Platform Migration using Wire, Claude Code and Rittman Analytics](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2FData-Pipeline-Diagram-selection-1-1.png%3Ffit%3D3840%252C1640%26ssl%3D1) Migrating an entire analytics platform including the data warehouse, data source extractions, semantic layer, BI reports, orchestration and reverse ETL syncs from one stack to another used to be the kind of project you hoped wouldn’t land while you were on the bench, because there was every chance you’d be asked to lead it. Large-scale data platform migrations are less about writing SQL than managing risk. Thousands of models, connectors, orchestration workflows, semantic objects, dashboards and reverse ETL syncs all need to be tracked, migrated, regression tested and reassembled into a platform that behaves exactly like the one it replaces. It’s the kind of work many engineers see as a chore rather than an opportunity to innovate, usually delivered under fixed-price contracts where keeping to the timetable and proving every change through exhaustive testing matters far more than moving fast or trying something new. A typical analytics platform you’d be asked to migrate would have one or more ingestion pipelines using technology such as Fivetran, Stitch or homegrown python scripts, loading data into a Snowflake data warehouse built-around a medallion-style architecture. Data transformations would be handled by dbt, orchestrated using dbt Cloud and data then synced into downstream CRM or other SaaS applications using reverse ETL technology such as Census or Hightouch. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2FData-Pipeline-Diagram-selection-1-2.png%3Fresize%3D3840%252C1640%26ssl%3D1) **The Perfect Use-Case for Agentic Delivery** Whilst a complex and thankless task for us humans to work-on, these types of projects are however a perfect use-case for AI-based agentic coding tools such as [Rittman Analytics’ Wire Framework](https://www.rittmananalytics.com/services/wire-framework). Why? Because unlike building a whole new platform or user application where users’ requirements can be interpreted in many ways and there’s no one “correct” answer to what they want, a data platform migration in the end just works, or it doesn’t. All of the individual Fivetran ingestions, dbt models, reports and reverse ETL syncs either provide the same output as before, or they’re defective and if they’re the latter, a coding agent can just fix and iterate and fix until it does get that same output, and then all of these migrated items outputting the same results as the originals adds-up to a completely migrated platform that’s functionally-identical to the one it’s replacing. And if there are tens, hundreds or thousands of these dbt models, reports or other migration inventory item to individually translate, test for equivalence and pass for migration, then technologies such as [Claude Code’s sub-agents](https://code.claude.com/docs/en/sub-agents) can chunk the work up into batches and parallel work streams, with your main Claude code agent orchestrating the process and reporting back to you when everything is done. The Wire Framework is Rittman Analytics’ AI-accelerated delivery system for data platform engagements. It uses **Claude Code** (Anthropic) or Gemini CLI (Google) as its runtime and encodes 20+ years of our data engineering and platform migration methodology as structured, executable agentic workflow specifications. Instead of one of our consultants’ manually writing dbt models, LookML, pipeline code or in this case, translated dbt models over several weeks, Wire directs the AI coding agent to produce all of these artefacts in a fraction of the time with embedded quality gates ensuring the output meets our standards. You can read more about Wire and our approach to agentic project delivery at: - [Wire Framework homepage](https://www.rittmananalytics.com/services/wire-framework) on rittmananalytics.com - [Wire Claude Code Plugin](https://github.com/rittmananalytics/wire-plugin) and [installation instructions](https://docs.rittmananalytics.com/en/latest/docs/getting-started/installation) - [Wire Documentation site](https://docs.rittmananalytics.com/en/latest/docs/intro) and [Data Platform Migration tutorial](https://docs.rittmananalytics.com/en/latest/docs/tutorials/platform-migration) - [Book a free discovery call with Mark Rittman](https://calendly.com/markrittman/initial-discovery-call-with-mark-rittman) to discuss a potential Wire project --- # Making Agentic Analytics More Accurate using Anthropic’s Agentic Data Stack and the Wire Framework Canonical: https://rittmananalytics.com/blog/2026/06/11/making-agentic-analytics-more-accurate-using-anthropics-agentic-data-stack-and-the-wire-framework URL: https://rittmananalytics.com/blog/2026/06/11/making-agentic-analytics-more-accurate-using-anthropics-agentic-data-stack-and-the-wire-framework Description: Anthropic published a blog article last week, “How Anthropic enables self-service data analytics with Claude”, describing an approach to one of the hardest problems in enterprise AI: how do you… Author: Mark Rittman Published: 2026-06-11T09:00:00+01:00 Modified: 2026-07-02T00:35:31+01:00 --- ![Making Agentic Analytics More Accurate using Anthropic’s Agentic Data Stack and the Wire Framework](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F360ff-0mllmr2anbsdy3fh2.png) Anthropic published a blog article last week, [“How Anthropic enables self-service data analytics with Claude”](https://claude.com/blog/how-anthropic-enables-self-service-data-analytics-with-claude), describing an approach to one of the hardest problems in enterprise AI: how do you get an AI agent to reliably answer business questions against a complex analytics platform? The problem will be familiar to anyone who’s connected Claude, Gemini or another LLM to Looker and asked something seemingly straightforward like: - “What was our revenue last month?” - “How many active customers do we have?” - “What’s our churn rate this quarter?” - “Which products generated the highest margin last month?” The answers usually come back fast, confident and well-formatted. The difficulty is knowing whether they’re actually correct. In most organisations, a mature semantic model contains dozens or hundreds of metrics, multiple date definitions, several customer counting methodologies and more than one reasonable way to calculate the same business concept. “Revenue” might mean booked revenue, recognised revenue, net revenue or gross revenue. “Active customer” might mean active in the last 30 days, active in the current billing period or currently subscribed. The AI agent has to select the right definition from several plausible alternatives and, when it gets that choice wrong, there’s often no obvious indication that anything has gone wrong. A coding agent such as Claude Code can be creative within boundaries defined by tests and documentation. The generated code either passes the tests or it doesn’t, providing an objective feedback loop. An analytics agent has no such luxury. Its job is to arrive at the one correct number and there is no compiler, test suite or runtime error waiting to tell it when the answer is wrong. In-essence, [Anthropic’s blog](https://claude.com/blog/how-anthropic-enables-self-service-data-analytics-with-claude) sets-out three underlying reasons why analytics agents struggle to come-up with consistently-correct answers: 1. **Concept-entity ambiguity:** A grown-up data model has hundreds of plausible fields, and the agent can’t reliably pick the ones that actually answer the question. Ask it for active users and it first has to decide what “active” even means (last login, last purchase, any event at all) and whether fraudulent accounts count, and what lookback window to apply. Nobody wrote those rules down, so it guesses. 2. **Staleness:** Sources, definitions and schemas move constantly, and the moment they do, your carefully-annotated model and the agent’s knowledge of it start drifting out of date and quietly handing back subtly wrong answers. 3. **Retrieval failure:** The right table might be sitting there, properly documented, and the agent simply doesn’t find it in a search space that big. The answer exists; it never gets to it. #### Anthropic’s solution: The Agentic Analytics Stack Anthropic’s fix is a four-layer stack, and the bottom two layers will look familiar to anyone who’s built a modern data stack over the past ten years: a **foundational data layer** of organised, curated, performant datasets and a **sources-of-truth** semantic layer that the business, via analytics agents and its BI tools, query against. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F360ff-0mllmr2anbsdy3fh2.png) What is new are the two new layers on top, and they’re built for agents rather than for human analysts. The data foundation and the source-of-truth layer beneath them shrink the field of plausible answers down until there’s a single governed one left standing. [**Skills**](https://www.rittmananalytics.com/blog/2025/10/27/from_prompts_to_skills_automating_financial_and_kpi_analysis_in_looker_with_claude_skills_and_mcp_toolbox) (the agent’s procedural knowledge of which sources to consult, in what order and what a finished answer is supposed to look like) make sure it actually finds and uses that governed answer rather than going off-piste. Skills are the key ingredient: without them, accuracy in Anthropic’s own testing never got above 21%; with them it sat consistently north of 95%. Contrast this approach with the one we’d been championing recently, providing example queries as ‘golden’ versions, which only added a few percent to accuracy in Anthropic’s own testing. **Validation** is how they catch what’s still slipping through; offline [evals](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) built from question-answer pairs that flag the critical gaps, ablation experiments that isolate one variable at a time and online checks like adversarial review, provenance footers and harvesting corrections as they come in. #### Anthropic’s Architecture Layered-Over the Modern Data Stack None of these four layers, apart from the naming of the last two, are new on their own. We’ve been building semantic layers, dbt governance, documented query patterns [for years](https://blog.rittmananalytics.com/how-rittman-analytics-does-analytics-modern-bi-stack-operational-analytics-using-looker-stitch-c5ea3df31d23), long before anyone came up with the concept of an LLM. What the Anthropic post does usefully is show how they behave differently when the thing consuming them is [an agent rather than a human analyst](https://www.linkedin.com/posts/tristanhandy_analyticsengineering-dataanalytics-aiagents-share-7452448030043299840-gdE4/) who can lean on their own judgement to paper over the gaps. Taking the concepts in the Agentic Analytics Stack and applying them to the architecture and data flow of a BigQuery, Looker and dbt modern data stack and you get something like the diagram below: ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2Fad3d6-1hjwrmxtlx4jnf7xp6m3mew.png) **Data foundations.** At the data warehouse and semantic model layer, the cure for concept-entity ambiguity is to have fewer tables, with better governance and control of sprawl, naming and duplication of names and concepts. If `total_revenue` resolves to one source instead of eight, most of the retrieval errors and hallucinations simply have nowhere to come from. In-practice that means deprecating near-duplicate models aggressively, dimensional modelling with the grain written down explicitly and clear and unambiguious labelling in-terms an LLM can easily-understand. **Sources of truth.** If you’re a Looker customer this layer already exists in that your LookML explores define the metrics, dimensions and joins in one place. The challenge is to make the agent route through them rather than around them, so that if a metric is defined in a LookML explore, the agent calls that definition instead of hand-rolling SQL against the raw tables. That single bit of discipline rules-out the largest category of wrong answers we see. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F27ff3-1_4qwyynl7d0-duvo28wasw.png) **Domain skills.** This is the bit that’s most directly liftable into our own work. Anthropic’s design approach writes a structured Markdown reference file per domain (which tables live there, what they mean, how they join, what the common questions look like) and crucially keeps those files in the same repository as the dbt code that builds the tables; doing that turns documentation from a separate workstream that’s permanently three months out of date into a code-review discipline. The knowledge skill then works as a router, narrowing the warehouse from hundreds of tables to the dozen that matter for a given question and the query-workflow skill encodes the steps, clarifying what’s being asked, checking the semantic layer, reading the domain reference, running it and sending it to adversarial review along with the provenance footer. **Validation.** The adversarial review sub-agent (a second Claude instance whose entire job is to challenge the answer before it reaches you) bought in Anthropic’s testing a 6% accuracy gain, at 32% more tokens and 72% more latency. Whether that trade is worth making is a per-client call, not a universal one. And arguably even more important is the offline eval harness: question-answer pairs per domain, run against every schema change through CI (continuous integration) test. Skip this and accuracy starts to drift, with Anthropic seeing theirs fall from 95% to 65% inside a month with no active maintenance. Provenance footers (the source tier the number came from, how fresh the data is, who owns the domain) don’t fix a wrong answer, but they let whoever’s reading it judge how far to trust it. A figure that came out of a governed LookML measure is a very different animal from one an agent improvised in SQL against an undocumented table, and the footer is what tells them which one they’re looking at. ### A Worked Example using the Wire Framework So how might this work in-practice? Let’s consider an example of a mid-size online fashion retailer, six years old, with their data platform built-on Snowflake with data loaded using Fivetran, modelled in dbt and surfaced in Looker. Their data warehouse got built incrementally, over many years and with a table added whenever somebody needed one. Metrics were defined team-by-team with little central co-ordination and no real way for users to understand which table to use and when. Their experience so far with connecting analytics agents such as [Claude to Looker via it’s new MCP Server](https://www.rittmananalytics.com/blog/2025/10/13/claude_meets_looker_building_smarter_connected_analytics_with_googles_mcp_toolbox) was that answers were correct around a third of the time; not because Claude couldn’t add-up or understand the question, but because their warehouse and semantic layer structure gave it too many ways to get the answer wrong. ### An Example Implementation using Wire Wire is our structured delivery system for data platform engagements, built on top of Claude Code and Gemini CLI and outlined previously in our blog [“Introducing the Wire Framework: The “Secret Sauce” Behind Our AI-Augmented Analytics Project Delivery”](https://blog.rittmananalytics.com/introducing-the-wire-framework-the-secret-sauce-behind-our-ai-augmented-analytics-project-7e7a2b50d9a3) Installed as a plugin for Gemini CLI or, as shown below, for Claude Code: /plugin marketplace add rittmananalytics/wire-plugin /plugin install wire@rittman-analytics /reload-plugins it encodes analytics engineering methodology as workflow specifications and covers the full delivery lifecycle from Discovery (Shape Up + RA Canonical SOP), Requirements, Design, Development, Testing, Deployment, Enablement, Platform Migration and now, Anthropic’s Agentic Data Stack. Wire is being used today by our team and our clients’ data teams to accelerate and extend their analytics development capability and you can read more about it using the two links below: - [Introducing the Wire Framework: The “Secret Sauce” Behind Our AI-Augmented Analytics Project Delivery](https://blog.rittmananalytics.com/introducing-the-wire-framework-the-secret-sauce-behind-our-ai-augmented-analytics-project-7e7a2b50d9a3) - [Build Data Platforms in Days, Not Weeks](https://www.rittmananalytics.com/services/wire-framework) - [Wire Plugin for Claude Code](https://github.com/rittmananalytics/wire-plugin) - [Wire Framework User Guide](https://github.com/rittmananalytics/wire-plugin/blob/main/USER_GUIDE.md) A new type of project (or “release”) that this toolkit supports is [Anthropic’s Agentic Analytics Stack](https://github.com/rittmananalytics/wire-plugin/blob/main/USER_GUIDE.md#18-running-an-agentic-data-stack-release), implementing its four-layer architecture more or less directly: governed canonical datasets, per-domain knowledge files sitting next to the dbt models, LookML-first routing through the Looker MCP server, adversarial review on every answer and an offline eval harness wired into CI. In this example we’d start first with the goals and scope for this agentic analytics layer implementation, written-down as a markdown document stored in the same github repo that contains the dbt and LookML code for their data platform. \# Statement of Work \## Background UK online fashion retailer on Snowflake + dbt + Looker. ~180 tables in the production analytics schema, built incrementally since 2019. A first Claude SQL agent managed ~35% accuracy on commercial questions. Root cause is governance: multiple revenue definitions across views, no canonical customer metric, at least two broken models still referenced by dashboards. \## Objectives 1\. Audit the dataset for governance maturity — duplicates, naming, ambiguity. 2\. Design a canonical model: one table per entity, deprecation schedule, tiering policy. 3\. Add canonical LookML measures per domain, grouped under Canonical Metrics. 4\. Generate per-domain DOMAIN\_REFERENCE.md files colocated with the dbt marts. 5\. Deliver an installable Wire skill: LookML-first routing via Looker MCP, provenance on every answer. 6\. Build an eval suite — 10+ Q&A pairs per domain, CI runner, per-domain thresholds. \## Scope In scope — five domains and their primary tables: Revenue orders\_fact, order\_items\_fact, returns\_fact, refunds\_fact Marketing campaign\_spend\_fact, campaign\_performance\_fact, attribution\_fact Customer customers\_dim, customer\_orders\_fact, customer\_lifetime\_value\_fact Inventory products\_dim, inventory\_snapshots\_fact, stock\_movements\_fact Operations fulfilment\_fact, carrier\_performance\_fact, zendesk\_tickets\_fact Out of scope: finance/ERP integration, wholesale channel, B2B accounts (Phase 2). \## Success criteria \- Aggregate accuracy \>\= 85% at launch against agreed question set \- Core commercial questions answered without analyst involvement \- Eval suite in CI catching regressions within 24 hours \## Warehouse Snowflake (production database: ANALYTICS), ~180 tables across 4 schemas, dbt Core on Snowflake adapter, Looker as BI tool with existing LookML project. \## Release type Use the Wire Framework agentic\_data\_stack release type lifecycle. #### Our Five-Phase Agentic Analytics Stack Delivery Workflow The `agentic_data_stack` release type runs a [fixed five-phase workflow](https://github.com/rittmananalytics/wire-plugin/blob/main/USER_GUIDE.md#command-sequence-1), designed to be run over an existing data platform implementation: ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F694fa-16-c9r3e5h9efd6rhgksgkg.png) **Phase 1: Audit.** Three parallel audits are run on that existing data platform: **dataset** (governance grade, duplication groups, deprecation candidates), **metric** (existing definitions, conflicts, coverage gaps) and **query** (stakeholder question patterns, LookML explore coverage). Output is three reports and a prioritised findings list, with a human review gate before design begins. **Phase 2: Design.** Two design artefacts: a governance design (canonical table designations, deprecation schedule with sunset timelines, tiering policy) and a semantic layer design (metric specifications, calculation definitions, grain documentation, conflict resolutions), and another human review gate before build begins. **Phase 3: Build.** Five sequential workstreams: canonical dbt model updates, LookML view and explore updates, per-domain `DOMAIN_REFERENCE.md` files colocated with the dbt marts, and an installable `SKILL.md` encoding the three-tier routing rule and hard-refusal list. Again, there’s a human review gate on each workstream before proceeding. **Phase 4: Validation.** An eval suite (question-answer pairs per domain with a CI runner and per-domain accuracy thresholds) and adversarial review configuration. And another human review gate before the launch gate runs. **Phase 5: Launch gate.** Running of per-domain accuracy validation against the SOW thresholds, where domains that clear proceed and domains that don’t loop back to the relevant build workstream are identified for a targeted fix. #### Phases 1 and 2: using Wire’s autopilot mode For demo purposes, we run the first two phases using [Autopilot](https://github.com/rittmananalytics/wire-plugin/blob/main/USER_GUIDE.md#21-wire-autopilot-autonomous-execution), an experimental feature in Wire that lets the agent take decisions that would normally be human-in-the-loop. /wire:autopilot docs/sow.md The audit phase would then grade the dataset and surface duplication groups. In a warehouse of this age and history, the sort of things we’d expect it to highlight might include: - **A broken attribution model.** `marketing_attribution_fact` references `session_touchpoints_fact`, which got renamed to `web_sessions_fact` eight months ago during a dbt refactor. The model builds and returns no rows. Ask anything about marketing attribution and you get back a confident zero. It goes onto the hard-refusal list immediately, no sunset window, and the dbt fix goes into the next sprint. - `**orders_fact**` **versus** `**orders_mart**`**.** Both cover the same order history, but `orders_mart` was built eighteen months ago to add return adjustments and has drifted since — it applies return deductions at the order level rather than the line-item level, so any question about revenue by product category produces different numbers depending on which table you happen to hit. `orders_fact` with the correct join to `returns_fact` is canonical; `orders_mart` goes on the 90-day sunset list. - **Duplicate customer metrics.** `customer_lifetime_value_fact` and `clv_summary` both exist, built by different analysts at different times, on different lookback windows and different return-adjustment logic. Neither is documented. The audit flags both for consolidation under a single canonical definition, to be settled in the design phase. - **Multiple revenue definitions.** Across the LookML project, “revenue” turns up in at least four forms — gross order value, net of returns, net of returns and discounts, and recognised revenue matching the finance team’s month-end close. Nothing anywhere records which one is authoritative for which purpose. #### The design review gate, where accuracy is actually determined Running Wire in Autopilot mode generates the governance design, proposes canonical table designations, deprecation timelines and metric definitions for all five domains and surfaces it at the review gate. And this is the point where letting Wire run in autonomous mode has to stop. The governance design resolves the metric conflicts the audit found using technical signals such as table age, column coverage, row-count consistency. That’s correct as far as it goes, but the metric definitions it proposes encode business decisions, and those need a different kind of input before any build begins. Take revenue, for example. The governance design can spot that four definitions exist and propose consolidating around one but what it can’t do is work out which one matches what the finance director actually signs off at month-end; whether recognised revenue excludes pending returns, how cancelled orders are treated, whether gift-card redemptions are in or out. That filter is a conversation with finance, not something the audit can derive on its own. So before we go past the design review gate, we’d run a working session with the domain owners: - For Revenue: which definition is canonical, and what exactly separates recognised from gross? - For Marketing: which attribution model does the team genuinely use to make budget decisions, and is it the one that’s actually in the data? - For Customer: what counts as an active customer? last purchase within 30 days, 90 days, any activity on the site at all? - For Inventory: how is stock-on-hand defined when items are in transit? - For Operations: what’s an on-time delivery when the carrier marks it delivered but the customer’s raised a ticket saying it never arrived? The output is a set of specific annotations against the governance design document with filter values confirmed, definitions pinned down and every conflict resolved with a named business owner against the decision. Only then would we run the command to record that review and its findings, the step before we start the build. /wire:governance-design-review This matters because of how the adversarial review layer works downstream. The adversarial sub-agent (a second Claude instance challenging every answer before it reaches the user) is a routing guard, and a good one. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2Fce5a6-1p3rtr5zgkhhor4tivbxrqa.png) It checks whether the answer used the correct source tier, steered clear of refused tables, applied the filters the domain reference says are required, and produced a number that’s at least plausible. What it can’t check is whether the filter value itself is right. Encode gross revenue where the business means recognised revenue, and the adversarial agent will happily wave through wrong answers as correctly sourced. Getting those definitions right at the review gate is the thing that makes the adversarial layer worth having later. #### Phases 3 to 5: Manually, one artefact at a time With the governance design confirmed by the domain owners, we’d come out of autopilot and run the build commands individually — each artefact reviewed before the next one gets generated, with the canonical models first: /wire:canonical-models-generate northbrook-fashion /wire:canonical-models-validate northbrook-fashion /wire:canonical-models-review northbrook-fashion The validate step runs dbt tests against the canonical model changes, while the review step presents the deprecation plan as concrete commitments: `orders_mart` gets 90 days; `marketing_attribution_fact` goes straight onto the hard-refusal list with no sunset window, because a silent zero-row return is worse than an honest error — at least an error stops someone acting on a made-up number. Now we generate the LookML canonical measures, one workstream per domain, updating the existing view files and adding a `Canonical Metrics` field group to each: /wire:lookml-views-generate northbrook-fashion /wire:lookml-views-validate northbrook-fashion Before reviewing, we’d check the generated LookML back against the domain-owner session notes. For Revenue: does `net_revenue_gbp` exclude returns at line-item level, using the logic the finance director signed off? For Customer: does `active_customers_30d` use the purchase-based definition rather than any old site visit? The LookML validator catches broken field references; the session notes catch wrong business logic. A correction at this stage is a one-line LookML edit, while the same correction after the eval suite is in CI costs you a sprint. Domain reference files and agent configuration then get generated together: /wire:knowledge-skill-generate northbrook-fashion /wire:agent-config-generate northbrook-fashion This generates five `DOMAIN_REFERENCE.md` files, the documents the agent reads before answering any question in a domain; for example: \# Domain Reference: Revenue & Finance \*\*Domain owner\*\*: Head of Finance (CFO sign-off) \*\*Last reviewed\*\*: 2026\-05\-14 \*\*Mart table\*\*: \`northbrook\_fashion.marts.fct\_revenue\` \*\*LookML explore\*\*: \`revenue\` (Looker instance: northbrook.looker.com) \*\*Refresh cadence\*\*: daily at 06:00 UTC (Shopify → Snowflake via Fivetran) --- ## Canonical measure definitions ### Net revenue Gross order value minus refunds, minus discount codes applied at checkout. VAT excluded. Currency: GBP. \*\*Required filter\*\*: \`order\_status NOT IN ('cancelled', 'pending\_payment')\` # Domain Reference: Revenue & Finance \*\*Domain owner\*\*: Head of Finance (CFO sign-off) \*\*Last reviewed\*\*: 2026\-05\-14 \*\*Mart table\*\*: \`northbrook\_fashion.marts.fct\_revenue\` \*\*LookML explore\*\*: \`revenue\` (Looker instance: northbrook.looker.com) \*\*Refresh cadence\*\*: daily at 06:00 UTC (Shopify → Snowflake via Fivetran) --- ## Canonical measure definitions ### Net revenue Gross order value minus refunds, minus discount codes applied at checkout. VAT excluded. Currency: GBP. \*\*Required filter\*\*: \`order\_status NOT IN ('cancelled', 'pending\_payment')\` \*\*Required filter\*\*: \`is\_test\_order = false\` \*\*LookML measure\*\*: \`revenue.net\_revenue\_gbp\` \*\*Grain\*\*: one row per order line item Do not use \`gross\_revenue\` unless the question explicitly asks for revenue before refunds. When a user asks "what is our revenue", always return net revenue. ### Recognised revenue Revenue recognised on the date of dispatch, not date of order placement. For orders placed before month-end but dispatched after, recognised revenue falls in the following month. \*\*LookML measure\*\*: \`revenue.recognised\_revenue\_gbp\` \*\*Date field to use\*\*: \`dispatched\_at\`, not \`created\_at\` If a user asks for "revenue in March" they almost always mean recognised revenue for March. Clarify if ambiguous. ### Refund rate Refunds as a percentage of gross revenue in the same period. Numerator: \`refund\_amount\_gbp\`. Denominator: \`gross\_revenue\_gbp\`. Do not use net revenue as the denominator — this inflates the rate. --- ## Terms the business uses and what they mean in the data | Business term | Maps to | Notes | |---|---|---| | "Revenue" | \`net\_revenue\_gbp\` | Always net of refunds and discounts | | "Sales" | \`gross\_revenue\_gbp\` | Gross — before refunds | | "Returns" | \`refund\_amount\_gbp\` | Includes partial refunds | | "Discount" | \`discount\_code\_amount\_gbp\` | Checkout codes only — excludes sale pricing | | "Sale pricing" | not in warehouse | Markdown from RRP is in Shopify product data, not in this mart | | "AOV" | \`net\_revenue\_gbp / order\_count\` | Always calculated from net revenue | --- ## Data not in the warehouse | Metric | Where to find it | Notes | |---|---|---| | Revenue targets / budget | Google Sheets (Finance team) | Not ingested. Direct users there. | | Contribution margin | Not calculated | COGS not available in Snowflake | | Revenue by channel (paid vs organic) | \`fct\_orders\` joined to \`dim\_attribution\` | Not in this mart — use the marketing domain reference | If a user asks whether revenue is tracking to target, tell them targets are not in the warehouse and direct them to the Finance team's Google Sheet. --- ## Ambiguous fields — handle with care \`dispatched\_at\`. Only use \`created\_at\` if the user explicitly asks for orders placed in a period. \*\*International orders\*\*: this mart includes all geographies. GBP conversion uses the exchange rate at order placement (Shopify rate). If a user asks for EUR or USD figures, tell them the mart stores GBP only. \*\*Marketplace orders\*\*: Northbrook sells a small volume via ASOS Marketplace. These orders are not in Shopify and are not in this mart. If revenue figures seem low relative to expectations, check whether the user is expecting marketplace revenue to be included. --- ## Routing rules for this domain 1. Always try the \`revenue\` LookML explore first via the Looker MCP server. The explore pre-applies the \`is\_test\_order = false\` and order status filters. 2. If the explore cannot answer the question (e.g. requires a join not modelled in LookML), fall back to \`marts.fct\_revenue\`. Apply the required filters manually — they are not automatic at the mart layer. 3. Do not query \`raw.shopify\_orders\` or \`raw.shopify\_order\_lines\` directly. These tables are on the hard-refusal list for revenue questions — they include test orders, cancelled orders, and contain gross amounts only. --- ## Adversarial review checklist for this domain Before returning any revenue figure, the adversarial sub\-agent should verify: - \[ \] \`is\_test\_order = false\` filter applied (or Looker explore used, which applies it) - \[ \] \`order\_status\` filter excludes cancelled and pending payment orders - \[ \] Date field used is \`dispatched\_at\` unless question specifies order placement date - \[ \] Currency is GBP (or conversion caveat included) - \[ \] Refunds are excluded from "revenue" responses (net, not gross) --- The key design choices worth noting for the article context: the "data not in the warehouse" section is what makes the adversarial review meaningful for questions like "are we tracking to target?" — without it, the agent would either hallucinate a target or return a confusing partial answer. The routing rules section tells the agent exactly which table is on the hard-refusal list and why, so adversarial review has a concrete rule to check against rather than having to infer it from schema inspection. We’d send each one to its domain owner for a quick read-through before running the review command; not a forensic audit, just a sanity check that the canonical explore names, the measure descriptions and the common question patterns match what they actually know to be true. The adversarial sub-agent reads these same files when it’s challenging answers, so if the Revenue reference encodes the wrong return-adjustment logic, the adversarial agent takes that wrong logic as its benchmark and the failure stays invisible until it surfaces in production, which is the worst possible place to find it. #### Grounding the eval suit in human-confirmed truth /wire:eval-suite-generate northbrook-fashion Autopilot generated question-answer pairs across the five domains, but the trouble is that those answers are derived from the same LookML definitions generated back in Phase 3. This means in-practice that what the [eval suite tests](https://github.com/rittmananalytics/wire-plugin/blob/main/USER_GUIDE.md#the-eval-suite-and-launch-gate), as generated by this Autopilot run, is whether the agent answers consistently with its own definitions and not whether those definitions are right against business reality. It can be perfectly self-consistent and uniformly wrong. So before running `/wire:eval-suite-review`, we’d ask the domain owners to contribute some anchor questions, three to five per domain where they know the right answer from a source they trust independently of the warehouse; for example: - “What was net revenue in March?” with the answer straight off the signed-off management accounts. - “What was our 30-day active customer count at the end of last month?” from the weekly commercial dashboard the team already lives by. - “What was average order value during the winter sale?” from the trading report. These become fixed points in the harness, questions the agent has to get right against external ground truth, not just against itself. The generated questions fill out the coverage; the anchor questions are the actual benchmark. We’d expect this to take an afternoon across all five domains. The domain owners are answering questions they already know the answers to and writing them down — not reviewing YAML or writing SQL. That’s the whole point; it’s their afternoon, not the data team’s week. ### How the Analytics Agent Now Processes Queries The Wire Framework’s Analytic Analytics Data Stack workflow then [builds the parts an agent needs](https://github.com/rittmananalytics/wire-plugin/blob/main/USER_GUIDE.md#what-the-release-delivers): canonical dbt tables with deprecation dates, LookML explores updated with the confirmed canonical measures grouped under “Canonical Metrics”, and `DOMAIN_REFERENCE.md` files reviewed by the domain owners. An installable `SKILL.md` then enforces three-tier routing where LookML explores via the Looker MCP first, canonical dbt marts when finer grain is needed and hard-refusal list for the tables that must never be queried. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F86b26-1sxe9by-vmzh4k21u1qdn_a.png) Following a handful of representative questions through the skill shows what all that governance work actually delivers for your organisation; for example: - **“What was revenue last month?”** Without governance: the agent might hit `orders_mart` and apply return deductions at order level, producing a number that doesn’t tie to the finance close, or it might go to `orders_fact` directly and pick up gross rather than net. With the skill: the first call goes to the `net_revenue_gbp` measure in the Revenue LookML explore, line-item return logic confirmed by finance, and `orders_mart` is on the never-query list. - **“How many active customers do we have?”** Without governance: any one of three different active-customer definitions scattered across the LookML project. With the skill: the `active_customers_30d` measure, defined as customers with a completed purchase in the last 30 days, sourced from `customers_dim`, with the exact lookback window signed off by the commercial team. - **“What’s our marketing attribution by channel?”** Hard stop. `marketing_attribution_fact` is on the hard-refusal list, so the skill returns an explicit error and explains the model is broken pending a dbt fix — rather than the confident zero that looks for all the world like a real answer. - **“What was revenue per active customer?”** A cross-domain question. The skill pulls `net_revenue_gbp` from the Revenue explore and `active_customers_30d` from the Customer explore as two scoped calls and divides them in the agent layer. It doesn’t go off and attempt a speculative join across the domain boundary. - **“What’s our return rate by product category?”** The skill routes to the `return_rate_by_category` measure in the Revenue explore, which joins `returns_fact` to `order_items_fact` and `products_dim` at line-item grain. Without that canonical measure, an agent hand-rolling the SQL would join at order level as often as not and produce category figures that are quietly wrong. Two things sit underneath every one of those answers. 1. The adversarial sub-agent runs a pre-flight checklist: did the answer touch a refused table, bypass a LookML explore where a measure already exists, drop a filter the domain reference says is required, or forget the time period? 2. And every answer carries a provenance footer: Data sources: \[Tier 1 / LookML\] Revenue explore — net\_revenue\_gbp Filters: order\_status \= 'complete'; return\_status != 'pending'; period March 2026 Caveat: excludes orders with open return requests (counted on resolution) Freshness: orders\_fact max(order\_date) \= 2026\-06\-06 #### The launch gate evaluates and reports accuracy The launch gate stage then runs the eval suite domain by domain and reports accuracy against the SOW thresholds. /wire:launch-gate-validate northbrook-fashion From running this process, all five domains will tend to clear 85% once the governance working sessions and the anchor questions are in place, though the first pass without those steps will almost always throw up at least one domain with a definition gap that needs a targeted fix before it clears. A typical result looks like this: | Domain | Pass rate | Threshold | |\---|---|---| | Customer | 91% | 85% | | Revenue | 90% | 85% | | Operations | 88% | 85% | | Marketing | 87% | 85% | | Inventory | 86% | 85% | Marketing at 87% is where we’d look first; Attribution questions have a habit of surfacing edge cases in the filter logic — whether brand spend is separated out from performance spend, whether view-through conversions are counted — that the working session ought to have nailed down but doesn’t always cover exhaustively. This would be a LookML measure correction rather than a data problem, and it goes into the follow-up sprint rather than blocking the gate. The overall shape of it though is from roughly a third of questions coming back usable to the high eighties across five domains, and the key takeaway is that almost all of the calendar time sits in the domain working sessions and the anchor-question review, not in the automated build steps. The mechanical work, through in our case our Wire Framework plugin to Claude Code, is fast and getting faster. The governance conversations are where the time goes, and they’re also where the accuracy comes from, and you can’t buy your way out of them with a better model. The [Anthropic post](https://www.anthropic.com/blog/how-anthropic-enables-self-service-data-analytics-with-claude) is well worth reading in full. The most valuable thing in it isn’t the architecture diagram, useful as that is; it’s the evidence that, for this class of problem at least, you get a far better return on money spent governing your data than on money spent chasing a cleverer model. Get yourself down to one revenue table and you’ll beat any amount of prompt engineering and if you’re starting from forty, no prompt on earth is going to save you. The `agentic_data_stack` release type is available in Wire Framework from v3.7.5; run `/wire:new`, select “Agentic Data Stack”, and you’re off. ### Interested? Find Out More! [Rittman Analytics](https://www.rittmananalytics.com/) is a boutique data analytics consultancy that helps ambitious, digital-native businesses scale-up their approach to data, analytics and generative AI. We’re authorised delivery partners for Google Cloud along with Oracle, Segment, Cube, Dagster, Preset, dbt Labs and Fivetran and are experts at helping you design an analytics solution that’s right for your organisation’s needs, use-cases and budget and working with you and your data team to successfully implement it. If you’re looking for some help and assistance with your AI initiative or would just like to talk shop and share ideas and thoughts on what’s going on in your organisation and the wider data analytics world, [contact us now](https://calendly.com/markrittman/initial-discovery-call-with-mark-rittman) to organise a 100%-free, no-obligation call — we’d love to hear from you! --- # Google Next 2026: What’s New for Looker, BigQuery, Data Platforms and Agentic Analytics Canonical: https://rittmananalytics.com/blog/2026/04/26/google-next-2026-whats-new-for-looker-bigquery-data-platforms-and-agentic-analytics URL: https://rittmananalytics.com/blog/2026/04/26/google-next-2026-whats-new-for-looker-bigquery-data-platforms-and-agentic-analytics Description: Lydia, Olivier Dupuis and I were representing Rittman Analytics at Google Next 2026 in Las Vegas this last week, and it certainly was an event at Google-scale. In a masterpiece… Author: Mark Rittman Published: 2026-04-26T09:00:00+01:00 Modified: 2026-07-02T00:35:25+01:00 --- ![Google Next 2026: What’s New for Looker, BigQuery, Data Platforms and Agentic Analytics](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F3358f-1vzv4jkdgans1-jnm34yfnq.jpeg) Lydia, [Olivier Dupuis](https://medium.com/u/780f7e26f132) and I were representing Rittman Analytics at Google Next 2026 in Las Vegas this last week, and it certainly was an event at Google-scale. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F3358f-1vzv4jkdgans1-jnm34yfnq.jpeg) In a masterpiece of great-timing I managed to twist my ankle hiking around Zion National Park the weekend before it all started, and so therefore spent the rest of the week slowly limping-around the vast event space trying to still get to see some of the great keynotes and breakout sessions relating to Looker, data analytics, data integration and AI & agentic analytics. This blog is actually Part 2 in a short series of Google Next 2026 updates from us, with Part 1 authored by Olivier Dupuis and titled [“Google Next ’26: Agents Everywhere, Semantics Nowhere”](http://blog.rittmananalytics.com/google-next-26-the-agent-stack-is-ready-the-semantic-engine-isn-t-44d1287e31f9). As you’ll see from my post I came-away with a slightly different take on the role of the semantic layer in Google’s new Agentic Data Cloud, so make sure you read Olivier’s post to for an alternative take on where LookML fits into Google Cloud’s grand strategy. That said, the overall framing from Google this year for data analytics and data platform is a shift from a “system of intelligence” (reactive BI that answers questions when asked) to a “system of action” (proactive agents that monitor metrics, summarise findings and take action on behalf of the business). Backing-up my theory ([Olivier Dupuis](https://medium.com/u/780f7e26f132)) that every new thing we see announced today for the modern data stack was part of my Oracle consulting toolkit twenty years ago, [Actionable intelligence (and Thomas Kurian’s enthusiasm for it)](https://www.rittmanmead.com/blog/2010/07/obiee-11gr1-action-framework-and-conditions/) is clearly nothing new, but this re-emphasis is still doing useful work because it changes what you should be scoping when you commission a Looker delivery project in 2026. This post picks-out the changes that will matter most to clients commissioning Looker or BigQuery work over the next twelve months, plus a few lessons from the customer sessions that were, for me, the most valuable parts of the conference. ### Google’s Agentic Data Cloud The keynote announcement was Agentic Data Cloud, Google’s answer to the problem where tools we built to help humans understand data aren’t designed for a world where AI agents act on that data autonomously, and is built-around three ideas: ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F535fc-1yx-qpfn8a2te16zdrdo0-w.png) 1. **Agents need context, not just data.** Pointing a powerful AI agent at a raw data lake and expecting it to figure out your business is a recipe for expensive mistakes. Agents need to understand what your data means; that “net revenue” excludes returns, that a particular metric is calculated differently across two business units, that a specific contract clause changes how a shipment should be routed. Google’s Knowledge Catalog is designed to capture and continuously maintain this institutional knowledge, so every agent that touches your data is working from the same governed, accurate understanding of your business. 2. **Agents need to act, not just report.** The shift Google is describing is from a “system of intelligence” which tells you what happened, to a “system of action” which does something about it. A practical example from the conference was that rather than a dashboard showing an order cancellation spike, an agent detects the spike, performs root cause analysis across logistics and sales data, identifies the distribution hub causing the problem, recommends a rerouting action, and sets up a monitoring agent to watch for when conditions improve. No human needed to ask the question. 3. **Your data doesn’t need to move.** One of the persistent frustrations with cloud data platforms has been the expectation that you migrate everything to one place before you get value. Google’s approach allows agents to query data wherever it lives – across Google Cloud, AWS, Azure, Snowflake, Databricks, or on-premises systems – without forcing expensive and time-consuming data movement. ### Looker as the Agentic Data Cloud “Experience Layer” In practical terms, the Agentic Data Cloud is the combination of BigQuery, Google’s Knowledge Catalog, Gemini and Looker as its “Experience Layer”, working as an integrated system rather than separate tools. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F04c57-15rznnqta2x2bjyaebjl-4w.png) The conversational analytics, dashboard agents, triggered workflows and other Looker announcements at Google Next 2026 are the user-facing expression of this architecture, the point at which business users and analysts start to experience the difference between BI that reports and BI that acts. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F8994d-1pozv3uyd-otvsvpt6l5i9g.png) Looker’s new Explore experience pushes that further, replacing the old field-picker-and-filters paradigm with a visual-first, drag-and-drop interface that feels much-more like Google Search or Google Docs (or dare I say it, Omni). Three new Gemini assistants are now embedded directly in Explore: an insight assistant for natural-language navigation, an expression assistant that writes Looker expressions from plain English, and a visualization assistant that picks the right chart type for what you’re asking. For those of us who build these things for a living, less of our delivery time will go on hand-holding around the explore visualization layer. More will need to go on getting LookML quality and semantic correctness right underneath, because the AI assistants are reasoning from what we’ve defined in the model. A sloppy semantic layer will produce sloppy answers at conversational scale. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2Fbdc89-1vjswc6im3mgn8epopzc4rq.png) Dashboard Agents are now GA, which means dashboards are no longer the static, broadcast-only artefacts we’ve been delivering for the last decade. Users can ask a dashboard questions, get summaries of what’s on the dashboard and what’s changed, and drill into the “why” behind a number conversationally. The most compelling enterprise reference case was YouTube’s, where partner managers ask questions about creator portfolios across fifteen merged data sources, all from inside what looks like a fairly conventional Looker dashboard. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2Fcfc98-14oce6hkfow9eg5ht_kyjdq.png) Looker Triggered Workflows go further again, moving the agents from passive responders to proactive monitors. They watch metrics continuously and alert teams when thresholds are crossed, with the near-term roadmap including agents that draft emails or post to Slack automatically. For operational analytics this is more useful than it sounds, and it’s a kind of work clients weren’t asking us for two years ago. A few of the other Looker announcements that caught my attention were: - **Looker Agents in Gemini Enterprise**: Agents built in Looker can now be published to Gemini Enterprise and managed alongside other enterprise agents. For clients investing in Gemini Enterprise, Looker stops being a standalone BI tool and becomes part of a wider AI strategy. - **Conversational Analytics API now GA**: With native Looker SDK integration, iframe embed support and an open-source reference implementation, clients embedding conversational analytics into their own applications now have a supported path. I’ve had several conversations stall over the past year because the API was preview-only, and that blocker has gone. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F35a9c-1cax4vgvhwlccj_9trjyftq.png) - **LookML Agent for VS Code (Preview)**: A VS Code extension that handles the full LookML lifecycle from natural language, sitting well alongside our own [Claude Code-based LookML development workflow](https://www.rittmananalytics.com/blog/2025/12/claude-meets-looker-mcp-toolbox). - **Looker MCP as an open standard**: Both a managed MCP server and open-source options through MCP Toolbox. As I wrote in [Claude Meets Looker](https://www.rittmananalytics.com/blog/2025/12/claude-meets-looker-mcp-toolbox), connecting Looker to AI assistants like Claude opens-up cross-explore analysis and agentic reasoning that goes well beyond what’s possible inside Looker on its own. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F4798f-19nra24hqksg-cusrkc4aha.png) - **Native semantic layer integrations**: Looker now reads BigQuery graph definitions and Snowflake semantic views natively, removing friction where existing semantic logic in BigQuery previously had to be recreated in LookML. - **Multi-explore agents**: Agents can now draw from up to five explores simultaneously. Yahoo’s implementation combines HR, finance, search and support data into single agents across multiple Looker instances. #### Practitioner Lessons from Yahoo and Virgin Media O2 The Yahoo and Virgin Media O2 conversational-analytics sessions were the most valuable parts of the conference for those delivering this kind of work. The takeaways are worth internalising if you’re rolling-out conversational analytics in Looker over the next twelve months. The headline learning is that golden queries and YAML-formatted agent instructions are the two biggest levers for accuracy. Yahoo’s switch from JSON to YAML in their agent instructions noticeably improved output quality, and they then moved the golden queries and business glossary into LookML rather than maintaining them separately, which significantly reduced instruction complexity because the LLM now reasons from the semantic layer rather than from hand-coded context. Yahoo’s summary, which I think is one of the more quotable lines from the conference, was that conversational analytics doesn’t start with the AI, it instead starts with the semantic layer. Get the data and the model ready, and conversational analytics is almost the next step in the evolution. What “good” LookML now looks like for conversational analytics has shifted accordingly. The new baseline includes labels, descriptions, synonyms and comments in every field definition, golden queries embedded directly in LookML, and a YAML-formatted business glossary in the model. Agent instructions are kept deliberately minimal because the semantic layer is doing the heavy lifting. Dashboard Agent configuration, triggered workflow setup and golden query development should now be baseline scope items in any serious set of Looker requirements, not optional extras. #### Data Studio (and the end of Studio in Looker) Looker Studio has, I kid you not, been rebranded back to Data Studio but now has an expanded mission. It now surfaces three artefact types: ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F5511a-1haiabgwey-ax4qgzlogn8q.png) 1. traditional reports and dashboards (sourced from BigQuery, Google and partner SaaS connectors or from the Looker semantic layer), 2. conversational analytics agents built in BigQuery and published into Data Studio, 3. and a new “data apps” concept where Colab notebooks are published as interactive applications. Data apps sit somewhere between a dashboard and a full application, and are worth understanding for clients who use Looker for operational reporting but want something more interactive without commissioning a full embedded application build. Also worth noting that with this rebrand back to Data Studio and the new self-service explores feature, the integrated version of Looker Studio (“Studio in Looker”) that was once the future of Looker’s front-end is no longer available for organizations that haven’t already enabled it, and is due to be [quietly taken-out the back and shot](https://docs.cloud.google.com/looker/docs/create-view-edit-reports) on June 15th, 2026. ### And What About Google BigQuery? A lot of what was previewed at last year’s Next has now moved through to GA, including the Data Engineering Agent, the Data Science Agent, Conversational Analytics in BigQuery, the Conversational Analytics API, the BigQuery ADK, the fully-managed remote BigQuery MCP Server, the open-source MCP Toolbox, and the BigQuery Agent Analytics Plugin for instrumenting agent performance telemetry. A couple of the new announcements that will matter more than they look at first glance: - **BigQuery Fluid Scaling**: BigQuery now auto-scales compute up when agents are active and back down when they aren’t, without manual sizing. The “cost of conversational analytics at scale” objection, which I’ve heard a lot from clients, weakens considerably. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F565bc-1e4icpkdtmf7jvqy7-ryh-a.png) - **BigQuery Graphs**: Native, zero-ETL graph capability scaling to billions of nodes with ISO GQL support. The more interesting announcement is “Measures in BigQuery Graph”, which combines governed metrics with relationship mapping so agents can reason through multi-hop dependencies. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F2f876-11atk5sf4dnwxy7gpkiksla.png) - **BigQuery Measures (as “Reasoning Extensions in Graph”)** : LookML measures govern *calculation* — defining what a metric means and ensuring consistent SQL generation across every analyst and agent using Looker. BigQuery Graph Measures govern *reasoning* — embedding those metric definitions into a graph of connected entities (products, suppliers, distribution hubs, routes) so agents can trace not just what a metric is but why it changed, by traversing the relationships between contributing factors. #### Data Analytics Migration Services Google’s BigQuery Migration Service has been rebranded as *Data Analytics Migration Services* and the upgrade is substantial. Anyone running a Snowflake-to-BigQuery, Cloudera-to-BigQuery or Databricks-to-BigQuery migration in 2026 should take a fresh look. The new capabilities include automated assessment for Cloudera and Databricks generating a migration blueprint with workload inventory and TCO estimates, compiler-level SQL translation from Hive, Snowflake and other dialects to GoogleSQL, lakehouse migration covering Iceberg, Parquet and Hive tables, and an inline agentic validation service with table-level and column-level accuracy scores. The migration APIs are MCP-accessible, which means there’s a clear path for using Claude Code or similar agentic tooling to drive a migration end-to-end. I’d been sceptical of the original BigQuery Migration Service for some of the more complex SQL translation work historically, but the new generation looks much better. The validation dashboard in particular replaces a lot of manual verification work that’s been dragging migration timelines for years. #### Cross-Cloud Lakehouse and Federation Google announced bi-directional read federation for Databricks Unity Catalog on AWS S3, and read federation for Snowflake Polaris and AWS Glue. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F6a859-1fcotjyxwbfgfh97lsacmva.png) All of it is built on the Apache Iceberg REST catalog standard, which means BigQuery can now query these external catalogs directly without data movement. For clients who can’t or won’t fully migrate, this changes the conversation. The pitch is no longer “migrate everything to BigQuery” but “keep your data where it is, query it from BigQuery, and put Looker and Conversational Analytics on top”. For organisations on Snowflake who’d otherwise be ruled-out of a GCP-first analytics practice this is a meaningful new option, and one I expect we’ll see used as a stepping-stone to fuller migrations. #### Migration Practitioner Lessons from Target, Wells Fargo and Virgin Media O2 The migration panel with Target, Wells Fargo and Virgin Media O2 produced several patterns worth sharing. They reflect lessons most of us have learned the hard way over the years, but they’re worth restating. The first and most important is to *move the semantic layer first*. Migrate the Looker or BI layer and give business users access to data on GCP before the underlying data engineering migration is complete, using tactical replication jobs to copy key datasets to BigQuery in the meantime. Virgin Media O2 has done this successfully across six large platform migrations and Wells Fargo endorsed the same pattern. A few other patterns worth holding onto: - *Reduce footprint before you migrate*. AI can identify unused and duplicate data before migration, shrinking what needs to move; one enterprise on the panel reported that more than 80% of their data was duplicated. - *Lift-and-shift, then modernise*. Refactoring mid-migration prolongs timelines unpredictably, so get to the new platform first and modernise there. - *Embed governance from day one*. Data contracts, business glossary and data quality rules should be established as part of the migration programme, not retrofitted afterwards when the political moment to do them properly has often passed. - *Start small on data contracts*. Identify a bounded area, make the supplier and consumer contracts neat for that use-case, and use it as the example of what good looks like before scaling. The “irrational goals” framing from Target was also worth borrowing. Define goals that are unachievable unless you rebuild with AI-native approaches, on the grounds that this changes the conversation from “how do we migrate” to “how do we transform”. #### Google Cloud Data Agent Kit Announced in preview, the Data Agent Kit is a VS Code extension and set of plugins for Gemini CLI, Cloud Code and Codex that packages skills and MCP tools for BigQuery data workflows. It includes agentic skills, MCP servers and tools for BigQuery access, and deep plugins for VS Code customisation, with the skills themselves open-source and customisable. For consultancies and client teams [such as ourselves](https://blog.rittmananalytics.com/introducing-the-wire-framework-the-secret-sauce-behind-our-ai-augmented-analytics-project-7e7a2b50d9a3?source=collection_home_page----206057218407-----0-----------------------------------) already building internal agentic delivery tooling, the Data Agent Kit is worth a serious evaluation. My early take is that it handles BigQuery-layer engineering work and is complementary to broader delivery orchestration and LookML, lining up well with the [Claude Skills approach](https://www.rittmananalytics.com/blog/2025/12/claude-skills-financial-analysis-looker) we’ve been using internally. ### Knowledge Catalog (formerly Dataplex) Dataplex has been rebranded as *Knowledge Catalog* and repositioned as the context backend for Google Cloud agentic workflows. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F5e6a9-1xolij9bkhrxkex8df6b3ma.png) It has three pillars: aggregation of metadata across Google services including Looker, partner catalogs (Atlan, Collibra, SAP and others), enrichment with automated metadata generation and a business glossary now GA and semantic search with a Context API for agents. The most practically instructive reference case was Bloomberg Media’s. Their “Knowledge Gap Agent” learns from agent failures, identifies where data context is wrong or incomplete, and proposes updates to Knowledge Catalog for human certification. The headline result was 102 usage notes auto-generated across twenty MVP tables, improving data access agent SQL accuracy by 63%. For most of our smaller engagements Knowledge Catalog will remain background context. For any client building conversational analytics agents at scale where institutional knowledge isn’t yet codified anywhere the agent can reach, it’s the missing piece, and Bloomberg framed this as the “42-dollar problem” : a category of failure that affects both human analysts and AI agents equally: the missing institutional knowledge that isn’t documented anywhere in the data platform. In Bloomberg’s example, a junior analyst correctly identified the right tables, wrote syntactically valid SQL, and applied standard net revenue logic but got the wrong answer because Bloomberg’s payments vendor charges a $42 fee for each customer dispute filed, which needs to be added to the revenue calculation. That rule isn’t in the schema, isn’t in the data pipeline, and wasn’t in any accessible documentation; instead it lived in a Confluence page from 2017 whose author had left the company. Olivier Dupuis argues in his [Google Next 2026 blog](https://odupuis.medium.com/google-next-26-the-agent-stack-is-ready-the-semantic-engine-isn-t-44d1287e31f9) that while Google is going all-in on the agent layer, its underlying semantic and business context is relatively weak. He suggests the Knowledge Catalog feels more like a metadata aggregator than a true enterprise operating system like OpenAI’s Frontier or Palantir’s Foundry. Without a strong semantic engine linking business meaning to agents, even advanced tooling risks producing outputs that drift, hallucinate, or correctly answer the wrong question. I agree with most of that diagnosis and with his opinion that the semantic layer matters, and it matters more in an agentic world than it ever did in a dashboarding one. Where I’d take a slightly more optimistic view is that the LookML semantic layer, as I noted in my [Conversational Analytics piece](https://www.rittmananalytics.com/blog/2025/05/conversational-analytics-data-agents-looker), already does most of what an Enterprise Operating System would need a business-context layer to do, eliminating the “multiple versions of the truth” problem, providing governed metric definitions, and serving as the trusted-metrics-store for AI assistants like Claude through MCP Toolbox. Knowledge Catalog being relatively quiet on the LookML integration front at this conference doesn’t mean LookML isn’t doing the work. It just means the work is being done one layer down from where Olivier is looking for it. The more concrete point is around what’s documented in Google’s [Best practices for configuring Conversational Analytics in Looker](https://docs.cloud.google.com/looker/docs/conversational-analytics-looker-best-practices). Conversational Analytics uses two primary inputs: the structure, fields, labels, descriptions and synonyms in your LookML model, and *distinct field values sampled from your data, with cardinality (i.e. the number of unique values) influencing how those values are used*. To pull those distinct values, Conversational Analytics uses the long-standing Looker API `model_fieldname_suggestions` method. This is the same mechanism that’s been powering filter suggestions in the field-picker and dashboard filters for as long as I’ve been working with Looker, now repurposed as a grounding signal for the LLM. Existing LookML parameters such as `suggestions`, `suggest_explore` and `suggest_dimension` become the developer’s way of actively shaping what values Conversational Analytics knows about. This pattern is one I’ve been hand-rolling for a while on my own BigQuery data warehouse. Alongside the standard `INFORMATION_SCHEMA`\-derived data dictionary, I maintain a markdown guide, exposed to Claude as a Skill and regenerated periodically by a profiling step, that describes each table with its row counts, date ranges and the value sets and cardinalities of the high-signal categorical fields. So Claude knows, for example, that my `messages_fact` table splits across iMessage (36K), Slack (17K), WhatsApp (1.1K) and so on, rather than having to guess. Conversational Analytics is now doing the same thing at the LookML layer, generalised across all Looker users rather than handcrafted for one dataset. To put that into LookML terms, if I had a `city` dimension populated from customer or operational data, Conversational Analytics would sample its distinct values, see “Brighton”, “Hove”, “London” and so on, and use cardinality to weight which values are likely to come up in user questions. The places to intervene as a LookML developer are with the `suggestions`, `suggest_explore` and `suggest_dimension` parameters, which look like this: view: customer { dimension: city { label: "Customer City" description: "The city associated with the customer's primary address. Most operational activity occurs in Brighton and Hove (UK head office) and London (UK secondary office)." type: string sql: ${TABLE}.customer\_city ;; # Hard-code a curated list of values rather than letting Looker # run SELECT DISTINCT against a large table. suggestions: \["Brighton", "Hove", "London", "Edinburgh", "Manchester"\] } dimension: customer\_segment { label: "Customer Segment" description: "Customer tier based on annual recurring revenue." type: string sql: ${TABLE}.segment ;; # Pull suggestions from another view rather than scanning # the much larger customer fact table. suggest\_explore: segments suggest\_dimension: segments.segment\_name } } Combine those parameters with descriptions, synonyms and a few golden queries embedded in the model, and you’ve got a LookML semantic layer doing exactly what Olivier is arguing the Enterprise Operating System needs to do. It provides grounded, governed business context to the agent layer, with cardinality-aware sampling of real values from your data, all driven from the same LookML powering your dashboards and explores. So my read on Knowledge Catalog versus LookML is a bit different. Today, LookML appears as just one input among many in the Knowledge Catalog architecture, and I can understand why that looks thin. I’d expect it to evolve significantly over the next twelve to eighteen months, as Google has every reason to make LookML the most prominent input into Knowledge Catalog rather than just one of several. The semantic engine that Olivier is rightly arguing the Enterprise Operating System needs isn’t quite missing. It’s mostly in LookML, and the open question is whether Knowledge Catalog grows-up to be the unifying business-context layer that surfaces LookML semantics across the whole agentic stack, or whether something else does. Either way, the bet on getting your LookML right is the bet most likely to age well. Where I do agree with Olivier without reservation is on the ontology question. None of what’s currently in LookML or Knowledge Catalog amounts to the kind of organisation-wide ontology that Palantir’s Foundry has been building for a decade or that OpenAI’s Frontier is now claiming to build, and that’s a genuine gap in the Google story Cloud Next ’27 will need to have a better answer to. But on the practitioner level, the work we’re doing today on LookML quality is the same work the Enterprise Operating System will eventually need to consume, whatever form it takes. ### What This Means for Looker and BigQuery Work in 2026 My most important takeaway from Next 2026 is, therefore, that the semantic layer is now the foundation for everything else in your Looker and BigQuery stack, including the work that’s nominally about AI rather than BI. If your LookML is right, the agents work, the conversational analytics gives sensible answers, and the dashboards’ agentic features add value rather than confusion. If it isn’t right, no amount of prompt engineering will fix it. The second takeaway is that the baseline definition of a Looker implementation has shifted. Dashboard Agents, triggered workflows, Conversational Analytics readiness and golden query development are now scope items rather than optional extras. If you’ve got a 2024-vintage scoping document you’re using to commission new work, it’s almost certainly going to under-deliver against what your business users will expect. The third is that cross-cloud federation and migration tooling has matured to the point where the architectural conversation with Snowflake-resident or on-prem-resident clients is different to where it was a year ago. Full migration is no longer the only option, and read federation across catalogs with Looker and Conversational Analytics on top is now a credible halfway house. For some clients it’s a better answer than a full migration project would ever be. ### Interested? Find Out More [Rittman Analytics](https://rittmananalytics.com/) is a boutique data analytics consultancy that helps ambitious, digital-native businesses scale-up their approach to data, analytics and generative AI. We’re authorised delivery partners for Google Cloud along with Oracle, Segment, Cube, Dagster, Preset, dbt Labs and Fivetran and are experts at helping you design an analytics solution that’s right for your organisation’s needs, use-cases and budget and working with you and your data team to successfully implement it. If you’re looking for some help and assistance with your AI initiative or would just like to talk shop and share ideas and thoughts on what’s going on in your organisation and the wider data analytics world, [contact us now](mailto:hello@rittmananalytics.com) to organise a 100%-free, no-obligation call. We’d love to hear from you! --- # Introducing the Wire Framework: The “Secret Sauce” Behind Our AI-Augmented Analytics Project… Canonical: https://rittmananalytics.com/blog/2026/02/25/introducing-the-wire-framework-the-secret-sauce-behind-our-ai-augmented-analytics-project URL: https://rittmananalytics.com/blog/2026/02/25/introducing-the-wire-framework-the-secret-sauce-behind-our-ai-augmented-analytics-project Description: If you’ve tried asking Claude or ChatGPT to generate a dbt model or a fact table definition for you, it’s impressive how fast it can produce syntactically-correct code. But building… Author: Mark Rittman Published: 2026-02-25T09:00:00+00:00 Modified: 2026-07-02T00:35:19+01:00 --- ![Introducing the Wire Framework: The “Secret Sauce” Behind Our AI-Augmented Analytics Project…](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F80274-1hdwqcjqntkn-ou-tbe3kwg.png) If you’ve tried asking Claude or ChatGPT to generate a dbt model or a fact table definition for you, it’s impressive how fast it can produce syntactically-correct code. But building a production data platform is more complicated than that as it’s a co-ordinated engineering effort where everything has to join and connect-up from start through to finish. We were early adopters of LLMs and agentic coding tools such as [Claude Code](https://docs.anthropic.com/en/docs/agents-and-tools/claude-code/overview) and [Gemini CLI](https://cloud.google.com/gemini/docs/codeassist/use-in-terminal) and the pattern we kept running into was what I’d call *context drift*. Each prompt is a fresh context window, and the model that generated your staging layer has no memory of the naming pattern it used when it then goes on-to build your warehouse layer; similarly, the LookML it writes references columns that it assumes exist in the dbt models it created earlier (but don’t), and the dashboard designs came-out of who knows where and bear no relation to the ones the client signed-off in the design phase. So we built the Wire Framework, originally developed around Claude Code by Olivier Dupuis and then extended and ported to Gemini Code Assist by myself. The best way to think about it is as an [exoskeleton](https://www.kasava.dev/blog/ai-as-exoskeleton) for analytics consultants: the human still makes every design decision and approval, but the mechanical work of translating those decisions into consistent, tested, documented code happens at machine speed. It’s become the agentic delivery engine behind our [AI-augmented delivery approach](https://rittmananalytics.com/services/ai-augmented-delivery) and you can get a sense of the impact it’s having already with this [review on LinkedIn](https://www.linkedin.com/feed/update/urn:li:activity:7430986316940529664/) from Power Digital’s [Scott Zakrajsek](https://www.linkedin.com/in/scozak/overlay/about-this-profile/), and our recent customer case study from [Barton Peveril College](http://rittmananalytics.com/case-studies/barton-peveril) in the UK. #### The Problem : Manual delivery works, but doesn’t scale A senior analytics engineer building a data platform manually follows a mental checklist refined over years of practice. They know that staging models should be named `stg___`. They know every primary key needs a `not_null` and unique test. They know that fact tables should reference dimension surrogate keys, not natural keys from staging. They know the client’s SOW deliverable D3 maps to three specific warehouse models. This works. It produces high-quality platforms. But it takes 15–20 hours of focused engineering time per project phase, and the methodology lives in the engineer’s head. When they leave the project, the conventions go with them. When a junior engineer takes over, they reinvent patterns — or worse, introduce inconsistencies that surface as bugs months later. #### Naive LLM generation is fast, but fragile Asking an LLM to “generate dbt models for our Salesforce and Stripe data” produces code quickly. But without persistent context across prompts, the output suffers from predictable failures: - **Naming**: Staging models named `salesforce_accounts` in one prompt, `stg_sf_account` in another - **Test coverage**: Primary key tests present on some models, missing on others. Foreign key relationship tests absent entirely - **Requirements traceability**: No connection between the dashboard tile and the SOW deliverable it implements - **Documentation**: Column descriptions present on the first model generated, absent by the tenth - **Convention drift**: `customer_id` in one model, `cust_id` in another, `account_pk` in a third The interesting thing is, these aren’t really knowledge failures — the LLM knows dbt conventions perfectly well if you ask it directly. They’re *context* failures. No single prompt can carry the full context of a multi-model, multi-phase data platform. #### The insight: encode the methodology, not just the prompts The thing is, the conventions and phase discipline that make manual delivery reliable aren’t really tacit knowledge — they’re documentable rules. A staging model naming pattern is a template. A test coverage requirement is a checklist. A phase gate is a prerequisite check. A review approval is a state transition. The Wire Framework encodes all of these as executable markdown specifications; documents detailed enough that an LLM can read them and follow them step by step, producing output that’s consistent, tested, traceable and reviewable. - **Naming consistency :** For manually-written code, it depends on the developer, and for naive LLM-generated code, it drifts across prompts. With the Wire Framework, it’s enforced by templates. - **Test coverage :** Manually-written code, it’s often incomplete. Naive LLM-generated code, it’s inconsistent. With the Wire Framework, it’s mandatory (every PK, every FK). - **Requirements traceability :** Manually-written, it’s informal. Naive LLM: none. Wire: every model maps to requirements. - **Documentation :** Manual: written after-the-fact. Naive LLM: degrades over session. Wire: generated alongside code. - **Knowledge retention** — Manual: lost with staff turnover. Naive LLM: not applicable. Wire: encoded in version-controlled specs. - **Timeline per phase** — Manual: 15–20 hours. Naive LLM: minutes (but requires rework). Wire: minutes of generation + 2–3 hours review. ### Architecture The framework is built on a strict separation between two components with different lifecycles — the filesystem structure below is for the Claude Code Plugin version of the framework. wire\-plugin/ ← Component 1: Plugin (stateless, replaceable) └── commands/dp/ ← Self\-contained command specifications (60 .md files) ├── start.md ← Entry point \- shows projects, suggests next action ├── new.md ← Delivery process definitions + project creation ├── status.md ← Project status reporting ├── requirements/ │ ├── generate.md ← Full workflow spec inline (200+ lines) │ ├── validate.md │ └── review.md ├── dbt/ │ ├── generate.md ← Full workflow spec inline (1,567 lines) │ ├── validate.md │ └── review.md ├── utils/ │ ├── atlassian\-search.md │ ├── jira\-create.md │ └── meeting\-context.md └── ... ← 18 artifact directories, each with generate/validate/review Your Repository/ └── .wire/ ← Component 2: Project Data (persistent, per\-project) └── 20260210\_meridian\_saas/ ├── status.md ← Source of truth (YAML frontmatter state machine) ├── execution\_log.md ← Append\-only audit trail of every command run ├── artifacts/ ← SOW, meeting notes, SQL examples ├── requirements/ ← Generated requirements spec ├── design/ ← Conceptual model, data model, pipeline design └── ... The Gemini CLI extension follows the same structure, with `.toml` files instead of `.md` and a `gemini-extension.json` manifest that also declares MCP server connections. - **Component 1** (Plugin) contains our specific delivery methodology encoded as workflow specification, template, convention, and validation rule in the form of self-contained command files. When a consultant selects a project type (`full_platform`, `pipeline_only`, `dbt_development`, `dashboard_extension`, `dashboard_first`, `enablement`) the framework knows exactly which artifacts to produce, in what order, and which gates each one must pass through. In Claude Code, commands are namespaced as `/wire:dp:` while the Gemini CLI extension uses .toml files and `/dp `. - **Component 2** (Project Data) is everything specific to a client engagement – the SOW, the generated artifacts, and most importantly the `status.md` file that tracks lifecycle state. When a consultant runs `/wire:dp:new` and selects a project type, the framework instantiates the corresponding process definition into `status.md` – writing the in-scope artifacts and their gate states as YAML frontmatter. From that point on, `status.md` is the running instance of the delivery process: it records which artifacts have been generated, which have passed validation, and which have been approved by stakeholders. This lives in the client’s repository under .`wire/`, is never touched by upgrades, and is shared across runtimes. What this means in practice is we can upgrade the framework mid-engagement without affecting project data. Update the plugin and all command logic is replaced instantly, but the project state stays exactly where it was. #### Self-contained command architecture When a consultant types `/wire:dp:dbt:generate 20260210_meridian_saas`, Claude Code loads the command’s .md file and reads its complete workflow specification: 1,567 lines covering prerequisites, upstream artifact reading, template application, SQL generation, test configuration, and status updates, and then executes it step by step. Each command file is both the discoverable entry point and the full executable specification. The `.md` file is pure markdown and the workflow specification itself: \# Generate dbt models \## User Input () $ARGUMENTS \## Workflow Specification \--- description: Generate dbt models following layered architecture \--- \# dbt Generate Command \## Prerequisites \[Checks status.md for approved data model...\] \## Step 1: Read Data Model Specification \[Reads upstream artifacts...\] \## Step 2: Generate Staging Models \[Templates, naming conventions, CTE patterns...\] \## Step 3: Generate Warehouse Models \[Dimension and fact table patterns...\] ...1,567 lines of complete workflow logic... In the Gemini CLI extension, the same workflow specification is wrapped in a `.toml`file that provides a description field and embeds the markdown in a prompt field. Modifying a command’s behaviour requires editing one file. There are no vector databases, no fine-tuned models, no RAG pipelines. Just markdown files, version-controlled in our internal plugin repository that propagate to every consultant on the next update. ### Core Innovations #### The Chain of Derivation The framework enforces a strict dependency chain where each artifact constrains the next: ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F80274-1hdwqcjqntkn-ou-tbe3kwg.png) Each arrow represents a *constraint transfer*. When `/wire:dp:data_model:generate` runs, it reads the approved conceptual model and pipeline design including the entity names, relationship cardinalities, source table mappings and replication architecture are all determined. The data model spec adds column-level detail (names, types, tests, grain) but cannot invent new entities or ignore upstream decisions. The `dashboard_first` project type follows an alternative chain where interactive dashboard mocks produce a visualization catalog that drives the data model directlym where the measures and dimensions the dashboards need determine what the warehouse must provide. Seed data enables dbt to run immediately, and a later data refactor step transitions from seeds to real client data. The constraint transfer principle is the same; only the source of the constraints changes. By the time `/wire:dp:dbt:generate` runs, every model name, every column name, every join path and every test is already specified in the data model. The AI’s job at this point is translation, turning the specification into SQL and not deciding what to build. This progressive narrowing of what the AI can choose to do is really the key to the whole thing. Each upstream artifact constrains the next, so by the time you get to code generation, there’s very little room for the kind of drift you’d normally see across LLM prompts. ### Three-Gate Lifecycle Every artifact passes through three gates: 1. **Generate** : The AI produces the artifact by reading upstream inputs, applying templates, and following conventions from the workflow spec. 2. **Validate :** Automated checks run against the generated output. Validation is severity-rated: *critical* failures block progression, *important* issues produce warnings and *info* items are suggestions. Example checks for dbt validation include naming convention compliance, test coverage verification, CTE structure validation and documentation completeness. 3. **Review** : A human stakeholder (the consultant, the client, or a technical lead) approves the artifact. Before presenting the artifact, the framework optionally retrieves context from Fathom meeting transcripts and Confluence documents, surfacing decisions and concerns from recent meetings. The reviewer can approve, request changes (with specific feedback recorded), or flag the artifact for workshop discussion. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F5b69f-10h1muvvbacypta1yisoquw.png) An artifact cannot, without manual override, progress downstream until all three gates are passed. The framework enforces this by having each generate command check `status.md` for prerequisite states: /wire:dp:dbt:generate reads status.md and checks: data\_model.generate \=\= complete ✓ data\_model.validate \=\= pass ✓ data\_model.review \=\= approved ✓ All prerequisites met → proceed with generation. If any prerequisite is missing, the command stops and tells the consultant exactly what to run next. Phase discipline is enforced by the AI reading project state, not by any external orchestration layer. ### Convention-First Code Generation The framework uses a two-tier convention system: 1. **Project-specific conventions** (highest priority): If a `.dbt-conventions.md` or `dbt_coding_conventions.md` file exists in the repository root, the framework uses it. This allows per-client customisation without modifying the framework. 2. **Embedded conventions** (fallback): The workflow specs contain comprehensive default conventions covering naming, SQL style, test coverage, documentation, and materialisation. Key conventions enforced across all projects include: - **Staging models**: `stg___` (e.g. `stg_stripe__subscription`) - **Dimension tables**: `_dim` (e.g. `customer_dim` - **Fact tables**: `_fct` (e.g. `revenue_fct`) - **Primary keys**: `_pk` (e.g. `customer_pk`) - **Foreign keys**: `_fk` (e.g. `customer_fk`) - **Timestamps**: `_ts` (e.g. `created_ts`) - **Booleans**: `is_` / `has_` (e.g. `is_churned`, `has_subscription`) - **CTE pattern**: `s_` prefix for source refs (e.g. `s_stripe_subscriptions`) - **Final CTE**: Mandatory `select * from final` - **Test coverage**: PK gets `not_null` + unique, FK gets relationships — every model, no exceptions These are not suggestions — the validation gate checks compliance before human review. #### The Status File as State Machine Each project’s status.md file serves dual purposes. Its YAML frontmatter is a machine-readable state machine that the framework reads and writes on every command. Its markdown body is a human-readable project dashboard. \# Machine-readable state (YAML frontmatter) artifacts: requirements: generate: complete validate: pass review: approved generated\_files: \["requirements/requirements\_specification.md"\] revision\_history: \- date: "2026-02-10" action: "generate" files: \["requirements/requirements\_specification.md"\] \- date: "2026-02-10" action: "review" result: "approved" reviewer: "Sarah Chen, VP Product" The revision\_history provides a complete audit trail — who approved what, when, and with what feedback. When a reviewer requests changes and the artifact is regenerated, the history shows the full path to approval including all intermediate feedback. There’s also defensive compatibility built in — workflow specs check for fields before using them, so a status.md created by an older framework version still works fine with newer specs. Missing fields are just treated as absent features, not errors. Complementing the structured YAML state machine, each project maintains an execution\_log.md — an append-only chronological log of every command that changes state. Each entry records a timestamp, the command invoked, its result, and a one-line summary. This provides a human-scannable narrative of the delivery process that is useful for consultant handovers, stakeholder auditing, and debugging failed commands. #### Optional Integrations via MCP The plugin pre-configures three Model Context Protocol (MCP) server connections that are available automatically to our consultants. In Claude Code, these are configured via the plugin manifest. In the Gemini CLI extension, they are declared in `gemini-extension.json`. **The** [**Atlassian (Jira + Confluence)** Remote MCP Server](https://www.atlassian.com/platform/remote-mcp-server) provides full project tracking. `/wire:dp:new` optionally creates a Jira hierarchy with one Epic per project, one Task per artifact, Sub-tasks for each lifecycle step (generate, validate, review). Every subsequent command syncs status to Jira and posts enriched comments showing file lists, revision numbers, and reviewer feedback. For example, after a data model review with changes requested: \*\*Data Model Design - Changes Requested\*\* \*\*Reviewer\*\*: James Park, Lead Data Engineer \*\*Date\*\*: 2026-02-13 \*\*Revision\*\*: 2 \*\*Feedback:\*\* \> Need to add a product\_plan\_dim to track plan tier changes over time. \> The current\_plan field on customer\_dim won't capture historical plan transitions. \*\*Previous reviews:\*\* \- 2026-02-12: Changes requested by James Park - "Missing subscription cancellation reason tracking" \- 2026-02-13: Changes requested by James Park \*\*Action\*\*: Address feedback, revise artifact, re-validate, and re-submit. Confluence is also searched during reviews — the framework queries for design documents, meeting notes, and technical specifications related to the artifact under review. Our [self-hosted **Fathom** MCP Server](https://github.com/sourcegate/mcp-fathom-server) on GCP Cloud Run provides meeting intelligence. Before every review command presents an artifact for approval, the framework searches Fathom for recent meeting transcripts mentioning the client and artifact type. It extracts and surfaces: - **Key decisions** made in previous meetings - **Outstanding concerns** raised by stakeholders - **Open action items** that may affect the artifact - **Feedback themes** recurring across multiple meetings This means the reviewer sees relevant meeting context *before* they make their approval decision, without anyone having to manually search through transcripts. Finally (for now), the [**Context7** MCP Server](https://context7.com/) provides up-to-date library documentation for development commands. When generating dbt models or LookML, the framework can query current documentation for packages like dbt-utils, avoiding hallucinated function signatures or deprecated syntax. * * * ### Walkthrough: A SaaS Customer Analytics Platform To show how this all works in practice, let’s walk through two stages in a complete engagement for **Meridian SaaS**, a B2B SaaS company that sells project management software to mid-market teams. They want to build a customer health scoring platform that combines data from three sources: - **Salesforce** (CRM): Accounts, opportunities, support cases - **Stripe** (Billing): Subscriptions, invoices, payment events - **Product database** (PostgreSQL): Feature usage events, login activity, workspace metrics The goal is to create a Looker dashboard showing customer health scores that combine product engagement, payment health, and support burden into a single composite metric, enabling their customer success team to prioritise outreach to at-risk accounts. #### Project Setup & Design Phase The diagram below shows the design phase of the process flow for the `full_platform` engagement required to meet the client’s needs. Each artifact passes through its gates (generate, validate, review) as described above and dashed arrows indicate iteration when a reviewer requests changes or validation fails, with the artifact returns to the generate step to cycle through the gates again. Solid arrows show the forward path when an artifact is approved. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2Fbdf04-1j0iaimv_nr1sudnc0yh4na.png) We start by creating a new project (or release, in a larger-overall engagement) by using the `/wire:dp:new` command (for Gemini Code Assist, we’d use `/dp:new`) /wire:dp:new The framework then asks a series of questions: project type, project name, client name, and SOW path. Selecting `full_platform` tells the framework to instantiate our full delivery process, activating all 14 artifact workflows across multiple phases (design through to enablement) and writing them into status.md as the project’s process definition. Because we’re on the main branch of our git repo, it creates a feature branch and switches us to that (handy, as I’m the worst for developing everything in main, just ask my team): You're on the main branch. A feature branch is required for project work. → Switched to branch: feature/20260210\_meridian\_customer\_health The framework then creates the project structure, and in the background creates a Jira project for me (or lets me map to an existing project and issues (tickets) if they’ve already been created). .wire/20260210\_meridian\_customer\_health/ ├── status.md ├── artifacts/ │ └── Meridian\_SOW.pdf ├── requirements/ ├── design/ ├── dev/ ├── test/ ├── deploy/ └── enablement/ Now we issue the command to generate the project requirements document. /wire:dp:requirements:generate 20260210\_meridian\_customer\_health To do this, Claude Code or Gemini reads the SOW PDF from `artifacts/,` extracts a set of structured requirements and maps each deliverable in the SoW into the `requirements_specification.md` artifact, splitting them into functional (below) and non-functional requirements: \## Functional Requirements \*\*FR-1\*\*: Customer Health Score Dashboard \- Composite health score (0-100) per account combining engagement, payment, and support signals \- Acceptance: Score correlates with known churned accounts at \>70% accuracy \*\*FR-2\*\*: Revenue Analytics \- MRR, ARR, expansion/contraction, net revenue retention by cohort \- Acceptance: Numbers match Stripe dashboard within 1% tolerance \*\*FR-3\*\*: Product Engagement Metrics \- DAU/WAU/MAU per account, feature adoption rates, workspace utilisation \- Acceptance: Metrics validated against product database event counts \*\*FR-4\*\*: Support Burden Indicator \- Open case count, average resolution time, escalation rate per account \- Acceptance: Matches Salesforce case report \*\*FR-5\*\*: At-Risk Account Alerting \- Automated identification of accounts with declining health scores over 30/60/90 day windows \- Acceptance: Customer success team receives weekly digest After validation (`/wire:dp:requirements:validate`) confirms completeness across all sections, the review command gathers context before presenting to the stakeholder: /wire:dp:requirements:review 20260210\_meridian\_customer\_health Before asking for approval, the framework searches Fathom for recent meetings and Confluence for related documents: \## Meeting Context from Fathom \*\*Meetings analysed\*\*: 3 meetings from 2026-01-28 to 2026-02-10 \### Key Decisions \- Agreed to use Stripe as billing source of truth (not Salesforce Opportunities) \- Pre-engagement call, 2026-01-28 \- Health score weighting: engagement 40%, payment 35%, support 25% \- Kickoff meeting, 2026-02-08 \### Outstanding Concerns \- "We need to make sure the health score accounts for seasonal usage dips in Q4" \- Sarah Chen, Kickoff meeting, 2026-02-08 \### Open Action Items \- \[ \] Meridian to provide Stripe API test credentials - Owner: Dev team, Due: 2026-02-14 \- \[ \] Confirm product database read replica access - Owner: Meridian DevOps The reviewer sees this context alongside the requirements document, then provides their decision: → Reviewer: Sarah Chen, VP Product → Status: Approved → Note: "Add a note about seasonal adjustment to FR-1 acceptance criteria" → Jira: MER-125 (Requirements \- Review) transitioned to Done The conceptual model captures the business entities and their relationships: /wire:dp:conceptual\_model:generate 20260210\_meridian\_customer\_health ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2Fcca6e-1zuszmc-08yea6y0xh-3cba.png) After approval, the data model specification defines every model in the dbt project, such as the `customer_health_fct` table from the warehouse layer. \### customer\_health\_fct \*\*Grain\*\*: One row per customer per month \*\*Surrogate Key\*\*: \`dbt\_utils.generate\_surrogate\_key(\['customer\_fk', 'health\_month'\])\` → \`customer\_health\_pk\` \*\*Foreign Keys\*\*: \- customer\_fk → customer\_dim.customer\_pk \*\*Columns\*\* (ordered: keys → dates → attributes → metrics → metadata): \- customer\_health\_pk (PK) \- customer\_fk (FK) \- health\_month\_ts \- engagement\_score (0\-100, based on DAU/feature adoption) \- payment\_score (0\-100, based on payment timeliness/failures) \- support\_score (0\-100, inverse of case burden) \- health\_score (weighted composite: 40% engagement + 35% payment + 25% support) \- health\_tier (derived: "Healthy" ≥75, "Monitor" ≥50, "At Risk" ≥25, "Critical" <25) \- is\_declining (boolean: health\_score dropped \>10 points in 30 days) \- dbt\_updated\_at \*\*Tests\*\*: \- unique(customer\_health\_pk) \- not\_null(customer\_health\_pk) \- not\_null(customer\_fk) \- relationships(customer\_fk → customer\_dim.customer\_pk) \- accepted\_values(health\_tier, \["Healthy", "Monitor", "At Risk", "Critical"\]) #### Development Phase With the data model approved, dbt code generation begins: /wire:dp:dbt:generate 20260210\_meridian\_customer\_health The AI reads the full data model specification and generates SQL models, YAML schemas, and source definitions, ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2Fc97fb-1suu47_li-1y78vmeotfabq.png) Example outputs in this phase are: **Staging model :** `models/staging/stripe/stg_stripe__subscription.sql`: {{ config( materialized='view', tags=\['staging', 'stripe'\] ) }} with s\_subscriptions as ( select \* from {{ source('stripe\_raw', 'subscriptions') }} where coalesce(\_fivetran\_deleted, false) = false ), renamed as ( select {{ dbt\_utils.generate\_surrogate\_key(\['id'\]) }} as subscription\_pk, customer as customer\_natural\_key, id as stripe\_subscription\_id, status as subscription\_status, cast(created as timestamp) as created\_ts, cast(current\_period\_start as timestamp) as period\_start\_ts, cast(current\_period\_end as timestamp) as period\_end\_ts, cast(canceled\_at as timestamp) as cancelled\_ts, items\_data\_0\_plan\_amount / 100.0 as mrr\_amount, items\_data\_0\_plan\_interval as billing\_interval, items\_data\_0\_plan\_id as plan\_id, case when status = 'active' then true else false end as is\_active, case when canceled\_at is not null then true else false end as is\_cancelled from s\_subscriptions ), final as ( select \* from renamed) select \* from final **Dimension model** : models/warehouse/core/customer\_dim.sql: {{ config( materialized='table', tags=\['warehouse', 'dimension'\] ) }} with s\_accounts as ( select \* from {{ ref('stg\_salesforce\_\_account') }} ), s\_subscriptions as ( select \* from {{ ref('stg\_stripe\_\_subscription') }} ), s\_usage as ( select account\_natural\_key, min(event\_ts) as first\_usage\_ts, max(event\_ts) as last\_usage\_ts, count(distinct event\_date) as total\_active\_days from {{ ref('stg\_product\_\_usage\_event') }} group by 1 ), enriched as ( select s\_accounts.account\_pk as customer\_pk, s\_accounts.salesforce\_account\_id as customer\_natural\_key, s\_accounts.account\_name as customer\_name, s\_accounts.industry, s\_accounts.employee\_count, s\_accounts.owner\_name as account\_owner, s\_subscriptions.subscription\_pk as current\_subscription\_fk, s\_subscriptions.plan\_id as current\_plan, s\_subscriptions.mrr\_amount as current\_mrr, s\_subscriptions.is\_active as has\_active\_subscription, s\_subscriptions.created\_ts as subscription\_start\_ts, s\_usage.first\_usage\_ts, s\_usage.last\_usage\_ts, s\_usage.total\_active\_days, s\_accounts.created\_ts, current\_timestamp() as dbt\_updated\_at from s\_accounts left join s\_subscriptions on s\_accounts.stripe\_customer\_key = s\_subscriptions.customer\_natural\_key and s\_subscriptions.is\_active = true left join s\_usage on s\_accounts.salesforce\_account\_id = s\_usage.account\_natural\_key ), final as ( select \* from enriched) select \* from final **Schema definition** : excerpt from `models/warehouse/core/schema.yml`: version: 2 models: - name: customer\_health\_fct description: \> Monthly customer health scores combining product engagement, payment behaviour, and support burden into a composite metric. Grain: one row per customer per month. columns: - name: customer\_health\_pk description: "Surrogate primary key (customer + month)" tests: - unique - not\_null - name: customer\_fk description: "Foreign key to customer\_dim" tests: - not\_null - relationships: to: ref('customer\_dim') field: customer\_pk - name: health\_month\_ts description: "First day of the health score month" tests: - not\_null - name: health\_score description: "Composite health score (0-100). Weighted: 40% engagement, 35% payment, 25% support." tests: - dbt\_utils.accepted\_range: min\_value: 0 max\_value: 100 - name: health\_tier description: "Human-readable health tier derived from health\_score" tests: - accepted\_values: values: \['Healthy', 'Monitor', 'At Risk', 'Critical'\] After generation, the automated validation gate runs: /wire:dp:dbt:validate 20260210\_meridian\_customer\_health \## dbt Validation Report \*\*Status\*\*: PASS \*\*Models\*\*: 12 (5 staging, 2 integration, 5 warehouse) \*\*Tests\*\*: 47 configured \### Naming Conventions | Check | Status | Count | |\-------------------------------|--------|-------| | Staging model naming | ✅ | 5/5 | | Warehouse fact naming | ✅ | 3/3 | | Warehouse dimension naming | ✅ | 2/2 | | Primary key naming (\_pk) | ✅ | 12/12 | | Foreign key naming (\_fk) | ✅ | 8/8 | | Timestamp naming (\_ts) | ✅ | 15/15 | | Boolean naming (is\_/has\_) | ✅ | 6/6 | \### Test Coverage | Check | Status | Count | |\-------------------------------|--------|-------| | PK: not\_null + unique | ✅ | 12/12 | | FK: relationships | ✅ | 8/8 | | Custom business rules | ✅ | 4/4 | | Total tests configured | ✅ | 47 | \### SQL Structure | Check | Status | |\-------------------------------|--------| | Final CTE present | ✅ | | Explicit join types | ✅ | | No bare SELECT \* | ✅ | | Source refs in s\_ CTEs | ✅ | \### Recommendations \- INFO: Consider adding freshness checks to Stripe source definitions The semantic layer translates the dbt models into LookML: /wire:dp:semantic\_layer:generate 20260210\_meridian\_customer\_health **LookML view :** `views/customer_health_fct.view.lkml`: view: customer\_health\_fct { sql\_table\_name: \`meridian-warehouse.analytics.customer\_health\_fct\` ;; dimension: customer\_health\_pk { primary\_key: yes type: string sql: ${TABLE}.customer\_health\_pk ;; } dimension: customer\_fk { type: string hidden: yes sql: ${TABLE}.customer\_fk ;; } dimension\_group: health\_month { type: time timeframes: \[month, quarter, year\] sql: ${TABLE}.health\_month\_ts ;; } dimension: engagement\_score { type: number sql: ${TABLE}.engagement\_score ;; value\_format\_name: decimal\_1 } dimension: payment\_score { type: number sql: ${TABLE}.payment\_score ;; value\_format\_name: decimal\_1 } dimension: support\_score { type: number sql: ${TABLE}.support\_score ;; value\_format\_name: decimal\_1 } dimension: health\_score { type: number sql: ${TABLE}.health\_score ;; value\_format\_name: decimal\_1 } dimension: health\_tier { type: string sql: ${TABLE}.health\_tier ;; html: {% if value \== "Critical" %} {{ value }} {% elsif value \== "At Risk" %} {{ value }} {% elsif value \== "Monitor" %} {{ value }} {% else %} {{ value }} {% endif %} ;; } measure: count { type: count drill\_fields: \[customer\_dim.customer\_name, health\_score, health\_tier\] } measure: average\_health\_score { type: average sql: ${health\_score} ;; value\_format\_name: decimal\_1 } measure: at\_risk\_count { type: count filters: \[health\_tier: "At Risk, Critical"\] } measure: healthy\_percentage { type: number sql: 1.0 \* ${at\_risk\_count} / nullif(${count}, 0) ;; value\_format\_name: percent\_1 } } **LookML explore** — explores/customer\_analytics.explore.lkml: explore: customer\_health\_fct { label: "Customer Health Analytics" description: "Customer health scores with account details, subscription, and revenue context" join: customer\_dim { relationship: many\_to\_one sql\_on: ${customer\_health\_fct.customer\_fk} \= ${customer\_dim.customer\_pk} ;; } join: subscription\_fct { relationship: many\_to\_one sql\_on: ${customer\_dim.current\_subscription\_fk} \= ${subscription\_fct.subscription\_pk} ;; } } #### Testing, Deployment & Enablement Phases With development complete and validated, the remaining phases test what’s been produced, deploy the dbt and Looker code and set-up the training and enablement for the client team. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2Fc4ebc-10jxoytdgximy9z4bn2qnsg.png) /wire:dp:data\_quality:generate 20260210\_meridian\_customer\_health generates additional data quality tests: source freshness alerts (Stripe webhook latency, Salesforce sync cadence), row count reconciliation between source and staging, and business rule assertions (health scores between 0–100, no orphaned subscriptions). /wire:dp:deployment:generate 20260210\_meridian\_customer\_health The framework generates two categories of enablement material — training session plans and technical documentation — tailored to the project’s specific artifacts and audience. /wire:dp:training:generate 20260210\_meridian\_customer\_health #### Final Status After all artifacts pass their three gates, the project status shows complete: | Phase | Artifact | Generate | Validate | Review | Ready | |\-----------------|------------------|----------|----------|--------|-------| | Requirements | requirements | ✅ | ✅ | ✅ | ✅ | | | workshops | ✅ | - | ✅ | ✅ | | Design | conceptual\_model | ✅ | ✅ | ✅ | ✅ | | | pipeline\_design | ✅ | ✅ | ✅ | ✅ | | | data\_model | ✅ | ✅ | ✅ | ✅ | | | mockups | ✅ | - | ✅ | ✅ | | Development | pipeline | ✅ | ✅ | ✅ | ✅ | | | dbt | ✅ | ✅ | ✅ | ✅ | | | semantic\_layer | ✅ | ✅ | ✅ | ✅ | | | dashboards | ✅ | ✅ | ✅ | ✅ | | Testing | data\_quality | ✅ | ✅ | ✅ | ✅ | | | uat | ✅ | - | ✅ | ✅ | | Deployment | deployment | ✅ | ✅ | ✅ | ✅ | | Enablement | training | ✅ | ✅ | ✅ | ✅ | | | documentation | ✅ | ✅ | ✅ | ✅ | #### **Engagement summary**: - 12 dbt models generated (5 staging, 2 integration, 5 warehouse) - 47 automated tests configured - 5 LookML views, 1 explore, 2 dashboards - 3 data sources integrated (Salesforce, Stripe, PostgreSQL) - Full deployment runbook with rollback procedures - 2 training session plans (data team hands-on workshop + end-user dashboard training) - Technical documentation with data model reference, health score methodology, and operations runbook - Complete audit trail in `status.md` with reviewer sign-offs at every gate, and `execution_log.md` documenting every command run with timestamps and results - Jira board fully synced: 1 Epic, 15 Tasks, 42 Sub-tasks — all transitioned to Done with enriched comments ### Timeline The Meridian SaaS engagement, from SOW to trained users, completed in about four weeks. The framework generated all artifacts in minutes per phase, and elapsed time was really spent on human review, client feedback cycles and iterative refinement. A comparable engagement built manually by a senior analytics engineer would typically require 8-12 weeks: 2–4 weeks for requirements and design, 2–4 weeks for dbt development and testing, 2 week for LookML and dashboards, and 1–2 weeks for deployment and enablement. To be clear, the framework doesn’t eliminate the need for engineering judgement as the consultant still makes design decisions, reviews generated output and handles client feedback. What it does eliminate is the mechanical work: writing boilerplate SQL, configuring tests, creating documentation and maintaining naming consistency across dozens of files. #### How is the framework licensed? When we use the Wire Framework on client engagements, we license it in a way that avoids lock-in after we’ve left. The version used on your project is delivered into your own repository, with full source code, and you receive a perpetual licence to use, modify and extend it for your internal business purposes. If you choose to bring the work in-house later or engage another partner, you’re free to do so. The framework is designed to sit alongside your dbt project, LookML models and documentation in a transparent and auditable way, not as a black box or hosted dependency. ### Interested? Find Out More! [Rittman Analytics](https://rittmananalytics.com/) is a boutique data analytics consultancy that helps ambitious, digital-native businesses scale-up their approach to data, analytics and generative AI. We’re authorised delivery partners for Google Cloud along with Oracle, Segment, Cube, Dagster, Preset, dbt Labs and Fivetran and are experts at helping you design an analytics solution that’s right for your organisation’s needs, use-cases and budget and working with you and your data team to successfully implement it. If you’re looking for some help and assistance with your AI initiative or would just like to talk shop and share ideas and thoughts on what’s going on in your organisation and the wider data analytics world, [contact us now](mailto:hello@rittmananalytics.com) to organise a 100%-free, no-obligation call. We’d love to hear from you! --- # So, Just How Relevant is Multi-Touch Attribution to Marketers in 2026? Canonical: https://rittmananalytics.com/blog/2026/01/28/so-just-how-relevant-is-multi-touch-attribution-to-marketers-in-2026 URL: https://rittmananalytics.com/blog/2026/01/28/so-just-how-relevant-is-multi-touch-attribution-to-marketers-in-2026 Description: If you’re running marketing for a B2C business, it’s somewhat of an understatement to say you’ve probably noticed that attribution has become significantly harder over the past few years. Between… Author: Mark Rittman Published: 2026-01-28T09:00:00+00:00 Modified: 2026-07-02T00:35:13+01:00 --- ![So, Just How Relevant is Multi-Touch Attribution to Marketers in 2026?](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2Ff030a-1tfrmp8woifje_y8pmms6gg.png) If you’re running marketing for a B2C business, it’s somewhat of an understatement to say you’ve probably noticed that attribution has become significantly harder over the past few years. Between browser privacy features, regulatory requirements, and rising ad-blocker usage, the tracking landscape we once relied upon has fundamentally changed. The much-discussed “death of the cookie” didn’t arrive as a single dramatic event. Instead, we’ve witnessed a gradual erosion of data quality that continues to accelerate. In this blog we’ll explore how attribution has evolved over the past few years and whether Multi-Touch Attribution still delivers meaningful value for your marketing decisions. ### Why marketing attribution is now so challenging Several converging forces have made traditional attribution approaches increasingly unreliable. Understanding these challenges is essential before we can explore practical solutions. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2Ff030a-1tfrmp8woifje_y8pmms6gg.png) - **Intelligent Tracking Prevention** remains one of the most significant obstacles. Safari and Firefox have led the charge here, with ITP detecting tracking behaviours and capping the lifespan of even first-party cookies to seven days — or just 24 hours in some scenarios. - **Chrome’s user choice model** has also reshaped the landscape. Rather than implementing a blanket ban on third-party cookies, Google now prompts users to make an informed choice about tracking. With opt-out rates mirroring what we’ve seen with Apple’s App Tracking Transparency, third-party data has become too fragmented for reliable individual-level tracking. Regulatory pressure continues to intensify as well. The Digital Markets Act now places the burden on “gatekeepers” like Google and Meta to ensure explicit consent. This has made tools like Consent Mode v2 a mandatory requirement rather than an optional enhancement. Crucially, the rise of ad blockers adds another layer of complexity: a substantial percentage of savvy users now use browsers like Brave or extensions that block tracking scripts entirely, creating what we might call “dark matter” in your analytics. #### What’s happened to cross-device tracking? Direct, cookie-based cross-device tracking is effectively dead for unauthenticated users. However, businesses have adapted using three primary methods. - **Deterministic matching** remains the gold standard. By encouraging users to log in across devices, you can [link their activity via a User ID](https://blog.rittmananalytics.com/stitching-identity-across-the-customer-journey-using-segment-google-bigquery-and-looker-6f026aed4fa1) in your CRM or CDP. This approach is completely unaffected by browser cookie restrictions, making it the most reliable option available. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F16697-0ctwwcmlxrojbxvdi.png) - **Conversion APIs** have become increasingly important. Platforms like Meta, TikTok, and Google now rely on server-to-server communication. Instead of a browser pixel firing, your server sends hashed first-party data — typically an email address — directly to the platform to match conversions to users. - **Probabilistic modelling** offers a fallback option. Using non-identifying signals like IP address, device type, and timing, these approaches estimate the likelihood that two sessions belong to the same user. While less accurate than deterministic methods, probabilistic matching provides directional insight when other options aren’t available. #### How server-side tracking can extend your attribution window Server-side tracking moves the data collection process from the user’s browser to your own server. This architectural shift offers several meaningful advantages. 1. First, it can bypass ITP restrictions. Because data is sent from your own domain (such as `data.yourdomain.com`), you can set “server-managed” cookies. These can extend your attribution window back to one year, bypassing the seven-day cap imposed on JavaScript-set cookies. For businesses with consideration cycles measured in weeks or months, this difference is substantial. 2. Server-side tracking also gives you greater data control. You can strip out sensitive PII before it ever reaches Google or Meta, ensuring higher compliance with privacy regulations. 3. Finally, reducing the number of tracking scripts in the browser improves page load speed — a factor that affects both SEO rankings and conversion rates. ### Understanding Consent Mode v2 As of 2026, Google Consent Mode v2 is no longer optional for businesses operating in the EEA or UK. It introduces two new signals — `ad_user_data` and `ad_personalization`—which dictate whether Google can use data for modelling and remarketing. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F4d3f3-0tvlvb_taiwwq1hfm.png) When users provide consent, you receive one-to-one data tracking just as before. When they don’t consent, GA4 uses behavioural and conversion modelling. The platform analyses the behaviour of users who did consent and uses machine learning to extrapolate patterns for those who didn’t. To implement this correctly, you must use a Google-certified Consent Management Platform to communicate these signals. When users don’t consent to being tracked, you still receive records of page views and other ecommerce and other user activity events in your GA4 or Segment data exports, you just can’t attribute them to a specific user or session. This does mean you can still measure the popularity of an article or product on your site, but you can’t connect that activity-together to understand the path to conversion. #### What about alternatives to GA4 such as Snowplow? While GA4 handles modelling as a “black box,” Snowplow (and the open-source [OpenSnowCat](https://opensnowcat.io/) fork from our [partners](https://rittmananalytics.com/partners/snowcatcloud) at SnowCatCloud) provides a warehouse-first approach that many data-mature organisations find compelling. With Snowplow and OpenSnowCat, you gain raw data ownership: granular, un-sampled event data delivered directly into your warehouse — whether that’s Snowflake, BigQuery, or another platform. You’re not constrained by Google’s attribution logic, which means you can build custom models using techniques like Markov Chain analysis with 100% of your first-party data. Snowplow also offers privacy by design. You maintain complete control over what’s collected, making it easier to comply with specific regional requirements. For organisations with sophisticated analytics capabilities, this flexibility can be transformative. #### What happens when a majority of users opt-out from tracking? When data loss reaches 40–60%, traditional MTA becomes unreliable. The industry has responded by adopting a triangulation approach that combines multiple methodologies. - **Modelling and extrapolation** uses the consented group as a proxy for the whole. While imperfect, this approach can still surface meaningful patterns when sample sizes remain adequate. - **Marketing Mix Modelling** offers a privacy-proof statistical method that correlates total marketing spend across channels with total sales, without requiring user-level tracking. MMM has experienced a renaissance precisely because it sidesteps the consent problem entirely. - **Incrementality testing** involves running holdout tests — for example, turning off ads in one geographic region — to measure the true lift a channel provides. This experimental approach delivers causal insights rather than correlational ones. ### Where does this leave multi-touch attribution? We’ve actually built two open-source attribution packages over the past few years and it’s useful to look-back at them to understand what’s still relevant in those packages, what is no longer relevant and how they can be updated and extended to go beyond basic multi-touch attribution. The first, [ra\_attribution](https://github.com/rittmananalytics/ra_attribution), is a dbt package for multi-cycle, multi-touch revenue attribution that was released four years ago when MTA was the undisputed standard. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F9846b-0jerwpzs0ust379ga.png) It handles the full complexity of attributing account openings, first orders, and repeat purchases; sourced data from Snowplow, Segment, or Rudderstack and supporting everything from First Click to Time-Decay models. The second, [ra\_attribution\_for\_ga4](https://github.com/rittmananalytics/ra_attribution_for_ga4), came about when Google removed rules-based attribution from GA4 entirely in 2023, forcing everyone onto their “data-driven” model. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F230cb-0qpnhzrroyzcbmzjk.png) A black box attribution model that changes weightings year-on-year makes comparison reporting essentially meaningless as you never know whether increased conversions came from your improved campaigns or Google tweaking the algorithm; this GA4-specific SQL and Looker repo used BigQuery export to give you back first-click, linear, time-decay and time-decay models using logic you control. #### What’s still relevant from those MTA packages 1. The [**multi-touch, multi-cycle logic**](https://blog.rittmananalytics.com/stitching-identity-across-the-customer-journey-using-segment-google-bigquery-and-looker-6f026aed4fa1) at the heart of ra\_attribution continues to matter. The assumption that users have distinct conversion cycles from account opening, first order to and repeat orders helps retailers understand how top-of-funnel awareness impacts initial purchases versus how retention marketing drives ongoing revenue. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2Fece6f-0jzmsi3r47e0cqtg6.png) This separation remains critical for allocating budget appropriately across acquisition and retention efforts. 2\. [**Identity stitching and blended user IDs**](https://blog.rittmananalytics.com/stitching-identity-across-the-customer-journey-using-segment-google-bigquery-and-looker-6f026aed4fa1) have if anything become more important, not less. The package’s use of a `blended_user_id` based on both visitor IDs and authenticated user IDs is now the modern standard. For B2C retailers where customers often browse on mobile and purchase on desktop, this logic is essential for de-anonymising the customer journey and connecting fragmented touchpoints. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2Fafd67-08uey3i9sgnrfdkih.png) 3\. The **warehouse-first architecture** has aged well. Whether you’re using ra\_attribution with Snowplow data in Snowflake or ra\_attribution\_for\_ga4 with BigQuery export, sourcing event data directly into your warehouse provides the raw data ownership necessary for deep attribution analysis. 4\. And [**support for diverse attribution models**](https://blog.rittmananalytics.com/stitching-identity-across-the-customer-journey-using-segment-google-bigquery-and-looker-6f026aed4fa1) from First Non-Direct Click to Time-Decay remains relevant as a tactical optimisation tool. Different models answer different questions, and having the flexibility to compare them side-by-side helps marketers understand channel contribution from multiple perspectives. The GA4 package in particular was built precisely because Google took this flexibility away. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F48e49-0ekxi7retsszaxhrl.png) #### What needs updating for 2026 Several assumptions that were however baked into both packages no longer hold true in 2026’s privacy landscape. - **Cookie-only lookback windows** present the most significant challenge. Both packages [default to a 30-day lookback window](https://blog.rittmananalytics.com/stitching-identity-across-the-customer-journey-using-segment-google-bigquery-and-looker-6f026aed4fa1), but with Safari’s ITP and Chrome’s user choice prompt, browser-set cookies are often capped at seven days or even 24 hours. Relying purely on browser cookies for a 30-day window will result in substantial data gaps and potentially missing the majority of touchpoints in longer consideration cycles. - **Lack of Conversion API support** is another gap. Both packages were designed around client-side event streams. Modern attribution requires integrating server-side Conversion APIs from Meta, Google, and TikTok to match conversions that browser-side tracking might have missed entirely. - The **omission of Marketing Mix Modelling** reflects the era in which we authored these packages. When individual-level tracking becomes fragmented beyond 40–50%, MTA alone becomes unreliable. Adding an MMM layer for top-down budget allocation is is something you’d want to implement after this package to create a more comprehensive attribution strategy. - Finally, the packages predate **Consent Mode v2**, which is now a strict regulatory requirement in the EEA and UK for any marketing features to function. Adding support for this into the package, in-terms of excluding activity that can’t be sessionised and linked to a single user would be required to ensure that session numbers and conversion rates weren’t incorrectly reported. ### So what’s our recommendation for MTA today? If you’re using either of our attribution packages. or building a similar warehouse-based attribution system then several updates will significantly improve their usefulness in 2026. 1. **Implement server-side GTM** to complement, and largely-replace, client-side user tracking. Moving to a server-side container on your own domain (such as `data.yourstore.com`) will enable you to set server-managed cookies that bypass browser caps and maintain that 30-day attribution window. This single change can dramatically improve data quality for both packages. 2. **Prioritise authenticated users aggressively.** Create compelling incentives early in the funnel — loyalty programmes, wishlists, saved carts — anything that encourages users to log in. The more authenticated users you have, the more the `blended_user_id` logic in ra\_attribution can deterministically stitch cross-device journeys without relying on probabilistic matching. For GA4 users, authenticated user IDs significantly improve the quality of cross-device reporting. 3. **Plan over time to move to hybrid attribution.** Use MTA logic for tactical, real-time campaign optimisation — understanding which creative is driving clicks today or which channels are contributing to this week’s conversions. Implement Marketing Mix Modelling separately for strategic, high-level budget decisions — determining whether your total spend on YouTube is actually driving incremental revenue or merely capturing demand that would have converted anyway. 4. **Expand beyond final conversions.** Rather than attributing only completed orders, track and attribute micro-conversions such as add-to-cart events, wishlist additions, or email sign-ups. This provides more data points for your models to work with in a low-consent environment where completed purchases may represent only a fraction of attributable events. 5. **Automate incrementality testing.** Use your dbt models to segment users for holdout tests. Stop ads in one geographic region or exclude a percentage of users from remarketing, then measure the true lift compared to a control group. These experiments provide causal evidence that no amount of correlational MTA can replicate. Multi-touch, multi-cycle marketing attribution still remains relevant in 2026, but it can no longer serve as your single source of truth. B2C retailers we’re working with today are treating MTA as one input among several: MTA for tactical optimisation, MMM for strategic allocation, and incrementality testing for causal validation. ### Interested? Find out more! Rittman Analytics is a boutique data analytics consultancy that helps ambitious, digital-native businesses scale-up their approach to data, analytics and generative AI. We’re authorised delivery partners for Google Cloud along with Oracle, Segment, Cube, Dagster, Preset, dbt Labs and Fivetran and are experts at helping you design an analytics solution that’s right for your organisation’s needs, use-cases and budget and working with you and your data team to successfully implement it. If you’re looking for some help and assistance with your AI initiative or would just like to talk shop and share ideas and thoughts on what’s going on in your organisation and the wider data analytics world, contact us now to organise a 100%-free, no-obligation call — we’d love to hear from you! --- # One Person Many Roles: Designing a Unified Person Dimension in Google BigQuery Canonical: https://rittmananalytics.com/blog/2026/01/26/one-person-many-roles-designing-a-unified-person-dimension-in-google-bigquery URL: https://rittmananalytics.com/blog/2026/01/26/one-person-many-roles-designing-a-unified-person-dimension-in-google-bigquery Description: When building our internal data warehouse, we quickly noticed that people and companies sit at the centre of most business processes and that the same person or company often plays… Author: Mark Rittman Published: 2026-01-26T01:16:57+01:00 Modified: 2026-07-01T23:49:43+01:00 --- ![One Person Many Roles: Designing a Unified Person Dimension in Google BigQuery](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F01%2F0.03577433557574594.png%3Ffit%3D1236%252C1044%26ssl%3D1) When building our internal data warehouse, we quickly noticed that people and companies sit at the centre of most business processes and that the same person or company often plays multiple roles over time. A consultant might be a staff member one day, a project lead the next and a contract signatory when closing a deal. Marketing contacts become clients, clients become referral partners and candidates sometimes become employees. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F01%2F5ceea-1qtoqcebqqm5jear3ijqwcw.png%3Fssl%3D1) While these roles share common attributes such as name, title and contact details, others are role-specific. It is tempting to model each role separately with dimensions for staff, contacts or candidates but this quickly leads to silos, duplication and the inevitable question of which table holds the correct email address. Instead, we modelled people in BigQuery using a single unified person dimension that captures every role a person plays across the business. #### The problem with role-specific tables Most analytics warehouses grow organically. You start with a CRM and build a contacts dimension. HR adds employee data so you create an employees dimension. Recruiting wants candidates, finance wants signatories and before long you are maintaining several tables that all store the same basic fields. This fragmentation causes predictable problems: - Duplicate records with conflicting data - Identity confusion across systems - Complex queries that require unions instead of joins - Higher maintenance as changes cascade across models - An incomplete view of how a person relates to the business over time Most importantly, it breaks the relationship history. When a prospect becomes a client, their marketing journey disappears. When a contractor becomes an employee, their project history is left behind. #### A unified approach using BigQuery STRUCTs We use Google BigQuery as our data warehousing managed service and instead created a single `persons_dim` table that accommodates every role a person could play in our business. **Uses boolean flags for common filters.** Queries like `WHERE is_staff = TRUE` are cleaner and faster than `WHERE staff_attributes IS NOT NULL`. **Stores current roles as an array.** The `current_roles` field enables queries like `WHERE 'PARTNER_CONTACT' IN UNNEST(current_roles)`. **Groups related attributes in STRUCTs.** Rather than 20 columns prefixed with `staff_`, nest them in `staff_attributes`. ``` CREATE TABLE `analytics.persons_dim` ( person_pk STRING NOT NULL, person_name STRING, person_phone STRING, all_emails ARRAY, person_linkedin_url STRING, is_staff BOOL DEFAULT FALSE, is_contractor BOOL DEFAULT FALSE, is_client_contact BOOL DEFAULT FALSE, is_partner_contact BOOL DEFAULT FALSE, is_prospect BOOL DEFAULT FALSE, is_candidate BOOL DEFAULT FALSE, current_roles ARRAY, primary_role STRING, staff_attributes STRUCT< job_title STRING, team_name STRING, employment_start_date DATE, weekly_capacity INT64, default_hourly_rate INT64, cost_rate INT64, manager_person_fk STRING >, crm_attributes STRUCT< lifecycle_stage STRING, lead_status STRING, lead_source STRING, conversion_event_name STRING >, candidate_attributes STRUCT< resume STRING, current_application_stage STRING, is_referred BOOL >, partner_attributes STRUCT< partner_company_fk STRING, is_primary_contact BOOL, has_signatory_authority BOOL > ); ``` **Uses bridge tables for contextual relationships.** A person’s role on a specific project or deal isn’t an attribute of the person — it’s a relationship. **Maintains a primary role.** We calculate `primary_role` based on a precedence hierarchy (staff \> contractor \> partner \> client \> prospect). ### How the person dimension joins to fact tables With a unified person dimension, every fact table that involves people can join through a single, consistent foreign key. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F01%2F00c0e-1g7pdnx6hikyticdiio9cng.png%3Fssl%3D1) This consistency transforms cross-functional analysis. Instead of building complex unions or maintaining role-specific join logic, every query follows the same pattern. ### Bridge tables for contextual roles Some roles however require more context recorded for them in-order for them to be useful in making decisions; a person isn’t just “a project team member” in the abstract; they’re a team member on *specific* projects with *specific* allocations. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F01%2Fa90a8-1akmwgq8vsnkwmblz6ungka.png%3Fssl%3D1) These bridge tables let us track the same person across multiple projects or deals with different roles on each. ## A unified person view for more effective messaging Looker, through its tight integration with Google BigQuery, makes it easy to import this unified person dimension table schema into your LookML project. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F01%2Fe43b1-17c22xeir1glqllxuxfml8a.png%3Fssl%3D1) The look below, for example, is filtered to only show people where `has_multiple_roles = Yes` and sorted by role count descending. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F01%2F2dcb8-1yxasxbrkz7fjlwy6p2amma.png%3Fssl%3D1) This helps us spot people like: - **Partners who are former staff,** and who would know our culture and working style intimately - **Clients who became referral sources,** our strongest advocates worth nurturing - **Prospects who are also community members,** who are already engaged with our content and forums such as the dbt Labs Slack - **Candidates who are also prospects,** who have a dual interest in working with/for us ### Altogether: a foundation for people analytics A unified person dimension such as this one transforms what’s possible with your analytics. The investment in consolidating person data pays dividends across every function as sales gets complete contact history, delivery sees full project context and finance can track the same individual from prospect through to paid invoice. ### Interested? Find out more! Rittman Analytics is a boutique data analytics consultancy that helps ambitious, digital-native businesses scale-up their approach to data, analytics and generative AI. We’re authorised delivery partners for [Google Cloud](https://rittmananalytics.com/partners/googlecloud) along with [dbt Labs](https://rittmananalytics.com/partners/dbt-labs), [Cube](https://rittmananalytics.com/partners/cube), [Dagster](https://rittmananalytics.com/partners/dagster), [Embeddable](https://rittmananalytics.com/partners/embeddable), [SnowCatCloud](https://rittmananalytics.com/partners/snowcatcloud) and [Fivetran](https://rittmananalytics.com/partners/fivetran). If you’re looking for some help and assistance with your data or AI initiative, [contact us now](https://calendly.com/markrittman/initial-discovery-call-with-mark-rittman) to organise a 100%-free, no-obligation call — we’d love to hear from you! --- # Why We’ve Tried to Replace Data Analytics Developers Every Decade Since 1974 Canonical: https://rittmananalytics.com/blog/2026/01/19/why-weve-tried-to-replace-data-analytics-developers-every-decade-since-1974 URL: https://rittmananalytics.com/blog/2026/01/19/why-weve-tried-to-replace-data-analytics-developers-every-decade-since-1974 Description: (This article was inspired by Stephan Schwab’s excellent piece “Why We’ve Tried to Replace Developers Every Decade Since 1969” which traces the recurring dream of eliminating software developers from COBOL… Author: Mark Rittman Published: 2026-01-19T20:49:55+01:00 Modified: 2026-07-01T23:54:18+01:00 --- ![Why We’ve Tried to Replace Data Analytics Developers Every Decade Since 1974](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F01%2F0.2706437466233763.webp%3Ffit%3D1360%252C768%26ssl%3D1) *(This article was inspired by Stephan Schwab’s excellent piece [“Why We’ve Tried to Replace Developers Every Decade Since 1969”](https://www.caimito.net/en/blog/2025/12/07/the-recurring-dream-of-replacing-developers.html) which traces the recurring dream of eliminating software developers from COBOL through to AI. Reading it, I recognised the same pattern playing out in my own field of data warehousing, data analytics and business intelligence; a fifty-year cycle of tools promising to democratise data work, each delivering genuine value while leaving the fundamental need for specialists stubbornly intact.)* ## The Pattern That Frustrates Everyone Every decade brings new promises: this time, we’ll finally make building analytics platforms simple enough that we won’t need so many specialists. From SQL to OLAP to AI, the pattern repeats. Business leaders grow frustrated waiting months for a data warehouse that should take weeks, or weeks for a dashboard that should take days. Data teams feel overwhelmed by request backlogs they can never clear. Understanding why this cycle persists for fifty years reveals what both sides need to know about the nature of data analytics work. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F01%2Faf653-188rk0igw4kkmr4vsshx7dw.jpeg%3Fssl%3D1) ### The Dream Was Born When Databases Learned English When IBM researchers [Donald Chamberlin](https://en.wikipedia.org/wiki/Donald_D._Chamberlin) and [Raymond Boyce](https://en.wikipedia.org/wiki/Raymond_F._Boyce) created [SEQUEL (later SQL)](https://en.wikipedia.org/wiki/SQL) in 1974, they had an explicit goal: create a query language that non-programmers could use. The vision was revolutionary; structured data could be retrieved using something approaching plain English. SELECT, FROM, WHERE, GROUP BY. The syntax was designed to be readable by business analysts, not just computer scientists. > “If we make the language readable enough, anyone who understands the data can query it themselves.” This vision had genuine appeal. Businesses were accumulating data faster than data analytics developers could write COBOL report programs. SQL promised to democratise data access, letting the people who understood the business retrieve the information they needed without submitting requests to the data processing department. What happened instead? SQL became another technical skill requiring specialised training. Business analysts who tried to write their own queries quickly discovered that readable syntax didn’t eliminate the complexity of joins, subqueries, aggregations or understanding why the results didn’t match last month’s report. A new class of “report writers” and “database analysts” emerged. The dream of eliminating specialised data analytics developers remained unfulfilled. ### The 1980s: OLAP Servers and Fourth-Generation Languages Will Let Business Users Model Their Own Data The 1980s brought a fundamentally different approach to the data access problem. Rather than making query languages more readable, pioneers like [John D.C. Little](https://firstanalytics.com/a-tribute-to-john-d-c-little-architect-of-modern-decision-science/) at MIT asked a different question: what if we stored data in the format that business users actually think about it? Little’s work on decision support systems had identified four criteria for computers that would support decision-making: robustness, ease of control, simplicity and completeness of relevant detail. His research project BrandAid, developed in 1975, evolved into a commercial product that would become known as [Express](https://www.orafaq.com/wiki/Express); a fourth-generation language and modelling system that spawned an entire industry. If you imagine a financial system, you tend to store your data by year, by line item, and possibly by branch; so what you have is a three-dimensional set of data. What the products were trying to do in those days was to put the data into the format and store it in the format that users understood it. The idea was: if you could put the data in the form they understood it, surely then it would be easier for them to get at it and do the analysis they wanted. Express Server, along with competitors like [TM/1](https://paristech.com/blog/cancelled-history-tm1/) and IBM’s System W, represented a genuinely different philosophy. These multidimensional OLAP (MOLAP) servers didn’t just make querying easier; they restructured how data was stored. Dimensions, hierarchies, aggregations and calculations were pre-computed and held in memory or specialised file structures optimised for analytical access patterns. The promise was compelling: business users, particularly in finance and planning, could build their own analytical applications. Essbase in particular was sold to power users, not to IT departments. An Essbase server could run under a finance analyst’s desk. The calculation language resembled spreadsheet formulas. Users who understood Excel could, in theory, build sophisticated planning and analysis applications without waiting for IT. And to a remarkable degree, it worked. These tools created genuine productivity for their target users. The number of true believers and evangelists amongst customers, amongst FP&A guys, was huge. They loved the product. It wasn’t waiting for IT to build a table and build a bunch of SQL queries. They owned it. Yet the dream of eliminating specialised data analytics developers still didn’t materialise. MOLAP applications required someone to design the dimensional structures. Someone had to understand how to handle sparse data, manage calculation order dependencies and optimise aggregation strategies. Complex business rules still required programming, even if the language looked like spreadsheet formulas. Performance tuning was an art form. Integration with source systems demanded ETL expertise. A new class of specialists emerged: the Express developer, the Cognos PowerPlay consultant, the TM/1 expert. The tools had changed; the need for people who deeply understood both the technology and the business had not. The MOLAP era taught the industry something important: you could restructure data to match how users thought about it, but the intellectual work of understanding the business well enough to structure it correctly; that remained irreducibly human. ### The 1990s: Data Warehouses and Dimensional Modelling Will Solve Everything The 1990s brought [Ralph Kimball’s dimensional modelling methodology](https://www.kimballgroup.com/) and the enterprise data warehouse movement. Building on the conceptual foundations that MOLAP had established, Kimball codified the approach for relational databases. The promise was compelling: build a properly modelled data warehouse with star schemas, conformed dimensions and consistent facts, and the complexity would be hidden from end users. > “Structure the data correctly once, and anyone can analyse it.” This was meant to democratise the benefits that MOLAP had delivered to finance departments, but for the entire enterprise. Facts and dimensions. Slowly changing dimensions. Conformed hierarchies. The complexity would live in the ETL layer, maintained by specialists, while business users enjoyed a simplified, denormalised view of their world through OLAP cubes and query tools. Organisations invested millions. Multi-year programmes launched to build enterprise data warehouses that would finally break the reporting bottleneck. Yet most data warehouse initiatives struggled or failed outright. Projects took years and cost fortunes. By the time the warehouse was “complete,” source systems had changed and requirements had evolved. The ETL pipelines became unmaintainable spaghetti. Most critically, building an accurate dimensional model required understanding the same business complexity that MOLAP development and report writing demanded. Where does a sale belong when the customer’s region changed mid-quarter? How do you handle late-arriving facts? What’s the grain of the transaction? The methodology was sound. The tools were capable. But the intellectual work of understanding business processes well enough to model them correctly; that couldn’t be automated away. Data analytics and data warehouse developers remained essential, and a new specialisation emerged: the ETL developer. ### The 2000s: Semantic Layers and Self-Service Reporting Will Free Business Users The 2000s saw two parallel movements, both promising to finally break the IT bottleneck. First, enterprise BI platforms matured with a powerful concept: the semantic layer. Business Objects’ “universe,” Oracle BI EE’s’ [“semantic layer”](https://docs.oracle.com/en/cloud/paas/analytics-cloud/acmdg/semantic-models-architecture.html) and similar abstractions promised to hide technical complexity behind business-friendly names. Data warehouse developers would build the semantic layer once, mapping cryptic table and column names to business terms. Then business users could build their own reports using familiar vocabulary (“Revenue,” “Customer,” “Product Category”) without knowing anything about the underlying SQL. Second, tools like Tableau and later Power BI began democratising visualisation itself. Drag a field onto a canvas, see a chart appear. No semantic layer configuration required. No IT involvement needed. “Self-service BI” became the goal of every MIS team lead. Both movements delivered genuine value. The semantic layer concept proved durable and important. Self-service tools genuinely lowered the barrier to creating visualisations. More people could explore data and build reports than ever before. Yet neither eliminated the need for specialists. Semantic layers required constant maintenance as source systems changed. Self-service created new problems: dozens of conflicting dashboards with different definitions of the same metrics. Performance issues. Security concerns. “Data chaos” became the term for organisations drowning in ungoverned self-service content. Meanwhile, something had been lost; the tools had democratised reporting, but in doing so, they’d lost something that the MOLAP era had provided: genuine analytical capability, the ability to model, calculate and explore. And the response to self-service chaos? A new wave of centralisation. Governed data sources. Certified datasets. Published data models. In other words: the work that data analytics developers had always done, now rebranded for the self-service era. ### The 2010’s: Analytics Engineering and the Promise That Analysts Could Do Their Own Data Engineering The mid-2010s brought a more fundamental reimagining of who should build data pipelines. Cloud data warehouses like [Amazon Redshift](https://en.wikipedia.org/wiki/Amazon_Redshift) (2012), Google BigQuery (2010, broadly adopted mid-decade) and Snowflake (2014) changed the economics and architecture of data warehousing. With virtually unlimited compute and storage available on demand, a new philosophy emerged: ELT instead of ETL. The insight was simple but powerful. Traditional ETL required specialised tools (Informatica, DataStage, SSIS) and specialised developers who understood their arcane interfaces and proprietary transformation languages. But if you loaded raw data into a cloud warehouse first, you could transform it using SQL. And analysts already knew SQL. > “If analysts can write SQL, why can’t they build their own data pipelines?” [dbt (data build tool)](https://www.getdbt.com/product/what-is-dbt), created by Fishtown Analytics in 2016, crystallised this vision into a product. Write your transformations as SELECT statements. Organise them into a directed acyclic graph of dependencies. Add tests. Generate documentation. Apply software engineering practices (version control, code review, CI/CD) to data transformation. No proprietary ETL tool required. No Java or Scala. Just SQL. The promise was intoxicating: the same analysts who understood the business well enough to write reports could now build the data models those reports consumed. The bottleneck of waiting for data engineers to build pipelines would dissolve. A new role emerged with a new title: the [“analytics engineer”](https://www.getdbt.com/blog/what-is-analytics-engineering); someone who combined the data engineer’s technical rigour with the analyst’s business understanding. And to a remarkable degree, it worked. dbt and the [“modern data stack”](https://www.getdbt.com/blog/future-of-the-modern-data-stack) genuinely lowered the barrier to building maintainable data transformations. Analysts who had been frustrated waiting for IT could now build their own staging tables, their own dimension tables, their own metrics. Data transformation became more accessible, more testable, more documented than it had ever been under the old ETL regime. Yet the dream of eliminating specialists proved, once again, elusive. Building a correct dimensional model still required understanding the business deeply. Deciding how to handle slowly changing dimensions still required judgment. Performance optimisation still required expertise. Managing dependencies across hundreds of models still required architectural thinking. And as dbt projects grew from dozens of models to hundreds to thousands, new problems emerged: sprawling DAGs, unclear ownership, conflicting definitions, the same “data chaos” that had plagued self-service BI, now manifesting in the transformation layer. The analytics engineer role didn’t eliminate the need for specialists; it created a new specialism. A very valuable one, to be sure. But still a role requiring deep expertise in both business logic and technical implementation. The tools had democratised access to building data pipelines, but the intellectual work of understanding what to build and why; that remained as demanding as ever. Semantic layers returned with renewed vigour in this era. Looker’s LookML. The dbt metrics layer. The promise: define business logic once, in code, and expose it consistently across all consumption tools. But building and maintaining a semantic layer still required someone who understood both the business definitions and the technical implementation. The tools were better. The work remained. ### AI and the Latest Chapter: What’s Actually Different This Time Today’s AI tools can do things that would have seemed like science fiction even five years ago. Claude Code can [scaffold a complete dbt package](https://blog.rittmananalytics.com/how-rittman-analytics-does-ai-augmented-analytic-delivery-and-the-story-behind-barton-peveril-a138eb6ec143) (staging models, dimension tables, fact tables, schema tests, documentation) in minutes rather than days. Feed it a DDL file and some requirements, and it will generate internally-consistent SQL that actually runs. Point a Looker MCP Server at your warehouse and an LLM can interrogate the semantic layer, discover available fields and generate dashboard definitions on demand. This is genuinely impressive. And it’s genuinely useful. But it’s worth being precise about what it changes and what it doesn’t. What AI handles well is the mechanical transcription of intent into code. Once you know you need a dimension table with SCD Type 2 handling, Claude Code can write the SQL and the dbt configuration faster than any human. Once you’ve decided which metrics belong on a dashboard and how they should be visualised, an AI can generate the LookML or the Tableau workbook. The typing, the syntax, the boilerplate; that’s where AI excels. What AI doesn’t handle is [the thinking that precedes the typing](https://rittmananalytics.com/case-studies/barton-peveril). Should this be SCD Type 2 or Type 1? The AI doesn’t know your business well enough to decide. What’s the grain of this fact table? The AI can’t attend your requirements workshop. How should “active customer” be defined when sales, marketing and finance all have different definitions? The AI will happily implement whichever definition you give it, but it can’t navigate the political and analytical trade-offs of choosing one. I’ve been [working on projects recently](https://rittmananalytics.com/case-studies/barton-peveril) where AI tools handle much of the code generation. The dbt models get written faster. The LookML views get produced more quickly. But the project still takes two weeks to deliver an MVP, not two hours. The time hasn’t disappeared; it’s been reallocated. Instead of spending those two weeks with developers heads-down writing SQL while stakeholders wait, then reviewing theoretical designs documented in Confluence, the time gets spent differently. More workshops with users. More iterations on working prototypes. More conversations about whether the dashboard actually answers the question they need to ask. The AI generates a first draft of the data model in an afternoon; the next ten days are spent refining it based on what users see when they actually interact with real data. This is a genuine improvement. But it’s not the improvement the “replace developers” dream imagines. The human expertise hasn’t been eliminated; it’s been concentrated on the parts of the work that were always the hardest and most valuable: understanding the business, making modelling decisions, translating vague requirements into precise definitions and iterating based on feedback. The bottleneck was never really typing speed. It was thinking speed. And thinking still takes time. What’s genuinely different this time is that the feedback loop can be faster. When generating code takes minutes instead of days, you can show stakeholders a working prototype sooner. When they say “that’s not quite what I meant,” you can regenerate and try again. The conversation shifts from “sign off on this specification document and we’ll build it over the next quarter” to “here’s a working version based on our conversation yesterday; what would you change?” That’s valuable. It might even be transformative for how analytics projects get delivered. But it’s transformation through faster iteration, not transformation through eliminating the need for people who understand data modelling. The AI generates the code. Humans still decide what code to generate. And that decision (what to build, how to model it, what questions it needs to answer) remains the hard part. ### Why the Dream Persists The recurring pattern reveals something important about how we think about complexity. Data warehousing and BI look like they should be simple because we can describe what we want in plain language. “Show me sales by region compared to last year.” That description sounds straightforward. The complexity emerges in the details. Which date defines when a sale happened: order date, ship date, invoice date or payment date? How do you handle returns processed in a different period than the original sale? What about currency conversion for international transactions? Should you use the exchange rate at transaction time or reporting time? How do you attribute revenue when a sales rep changes territory mid-quarter? What happens when organisational hierarchies are restructured; do you restate history or not? Each answer leads to more questions. The accumulated decisions, edge cases and business rules create genuine complexity that no tool or language can eliminate. Someone must think through these scenarios. That thinking is data warehouse and BI development, regardless of whether it’s expressed in SQL, Express 4GL, LookML, dbt or a prompt to an AI. ### The Constraint That Never Changes This fifty-year pattern teaches us something fundamental about data work itself. If the problem were primarily mechanical (too much SQL, too complex transformations, too many steps) we would have solved it by now. SQL made querying readable. MOLAP eliminated the need to think in tables and joins. Semantic layers hid technical complexity. Self-service tools eliminated IT gatekeepers. Analytics engineering democratised pipeline development. AI can now generate entire data models from descriptions. Each advancement addressed a real friction point. Yet the fundamental challenge persists because it’s not mechanical. It’s intellectual. A data warehouse is a model of a business. Building an accurate model requires understanding the business deeply; its processes, its edge cases, its exceptions, its evolving definitions. A BI solution is a lens through which people make decisions. Building an effective lens requires understanding what decisions matter, what questions lead to action and what context is needed to interpret the numbers correctly. You can’t shortcut that understanding any more than you can shortcut the understanding required to design a building or diagnose a medical condition. Better tools help. Experience helps. But someone must still think it through. ### What This Means for Data Leaders Understanding this pattern changes how you evaluate new tools and approaches. When someone promises that their platform will let business users build data solutions without DW/BI specialists, you can appreciate the aspiration while maintaining realistic expectations. The right question isn’t “Will this eliminate our need for data warehouse and BI developers?” The right questions are: - Will this help our data team work more effectively on complex problems? - Will this enable us to deliver certain types of solutions faster? - Does this reduce time spent on repetitive work so developers can focus on high-value modelling challenges? - Will our team need to develop new skills to use this effectively? - Does this tool require a well-governed semantic layer to function correctly; and do we have one? These questions acknowledge that DW/BI work involves irreducible complexity while remaining open to tools that provide genuine leverage. ### The Dream Serves a Purpose Perhaps the recurring dream of replacing data analytics developers isn’t a mistake. Perhaps it’s a necessary optimism that drives tool creation. Each attempt to make data work more accessible produces tools that genuinely help. The dream doesn’t come true as imagined, but pursuing it creates value. SQL didn’t let executives query databases themselves, but it did create a more accessible foundation than COBOL. MOLAP tools like Express and Essbase didn’t eliminate the need for data expertise, but they gave finance teams genuine analytical power they’d never had before; and created products so beloved that customers were still running unsupported versions decades after decommissioning. Semantic layers didn’t make metric definition trivial, but they made it more maintainable. Self-service BI didn’t remove IT from the equation, but it expanded who could explore data. Analytics engineering didn’t let analysts replace data engineers, but it gave them tools to contribute more directly to the data platform. AI won’t replace data analytics developers either. But it might finally let them spend most of their time on the work that actually requires human judgment (understanding businesses, making modelling decisions, having conversations with stakeholders) instead of the mechanical transcription of those decisions into code. That’s not the fulfilment of the fifty-year dream. It’s something more realistic and possibly more valuable: tools that let experts be more expert, faster. ## Moving Forward with Clear Eyes The next wave of data development tools will arrive. Some will provide genuine value. Some will repeat familiar promises with new technology. Having perspective on this recurring pattern helps you engage with new tools productively. Use AI assistants. Evaluate semantic layer platforms. Experiment with new transformation frameworks. But invest primarily in your people’s ability to think clearly about business complexity and data modelling. That capability remains the constraining factor, just as it was when Chamberlin and Boyce designed SQL to be readable by business users, and when John D.C. Little built BrandAid to let marketers analyse their own data. The organisations that thrive aren’t those waiting for tools to eliminate the need for data platform development expertise. They’re the ones investing in people who can think clearly about business complexity; and giving them the best tools available to express that thinking efficiently. Understanding this doesn’t mean rejecting new tools. It means using them with clear expectations about what they can provide and what will always require human judgment. Fifty years of tool evolution. Fifty years of promises. Fifty years of the same fundamental truth: data warehouses and BI solutions are business understanding made queryable. The dashboards and reports we create are the visible outcome of invisible reasoning about what questions matter, how to measure the answers accurately and what the numbers actually mean. Better tools help us express that reasoning more efficiently. They don’t eliminate the need to do it. ## INTERESTED? FIND OUT MORE Rittman Analytics is a boutique data analytics consultancy that helps ambitious, digital-native businesses scale-up their approach to data, analytics and generative AI. We’re authorised delivery partners for Google Cloud along with Oracle, Segment, Cube, Dagster, Preset, dbt Labs and Fivetran and are experts at helping you design an analytics solution that’s right for your organisation’s needs, use-cases and budget and working with you and your data team to successfully implement it. If you’re looking for some help and assistance with connecting Claude to Looker or would just like to talk shop and share ideas and thoughts on what’s going on in your organisation and the wider data analytics world, [contact us now](https://calendly.com/markrittman/initial-discovery-call-with-mark-rittman?utm_source=medium.com&utm_medium=blog&utm_campaign=customer_segmentation) to organise a 100%-free, no-obligation call – we’d love to hear from you! --- # How Rittman Analytics uses AI-Augmented Project Delivery to Provide Value to Users, Faster Canonical: https://rittmananalytics.com/blog/2026/01/19/how-rittman-analytics-uses-ai-augmented-project-delivery-to-provide-value-to-users-faster URL: https://rittmananalytics.com/blog/2026/01/19/how-rittman-analytics-uses-ai-augmented-project-delivery-to-provide-value-to-users-faster Description: Our latest case study features Barton Peveril College in Eastleigh, UK and the AI-augmented delivery approach we used to ship their Looker and Google Cloud MVP platform in December 2025. Over the past year we’ve quietly… Author: Mark Rittman Published: 2026-01-19T20:48:42+01:00 Modified: 2026-06-30T19:36:00+01:00 --- ![How Rittman Analytics uses AI-Augmented Project Delivery to Provide Value to Users, Faster](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F01%2F0.11539852215881174.webp%3Ffit%3D1400%252C545%26ssl%3D1) Our [latest case study](https://rittmananalytics.com/case-studies/barton-peveril) features [Barton Peveril College](https://barton-peveril.ac.uk/) in Eastleigh, UK and the [AI-augmented delivery approach](https://rittmananalytics.com/services/ai-augmented-delivery) we used to ship their Looker and Google Cloud MVP platform in December 2025. Over the past year we’ve quietly changed how we deliver analytics projects — not by replacing consultants with AI and not by promising instant dashboards at the click of a button. Instead, we’ve used tools like [Claude Code](https://code.claude.com/docs/en/overview) and [Google Gemini](https://gemini.google.com/app) to remove friction from the parts of delivery that slow teams down and divert attention from stakeholders and end-users. This post explains what has changed, what hasn’t and why it matters for teams building analytics platforms today. ### The problem with traditional analytics delivery Most analytics projects fail or stall for predictable reasons. Requirements take too long to surface. Early designs are abstract and hard to react to. Delivery teams spend weeks translating intent into code before anyone sees value. A typical engagement might involve two weeks of requirements gathering, another two weeks of data-modelling design, then four to six weeks of build — potentially ten weeks before stakeholders see a working dashboard. By then the business has often moved on. The problem isn’t technical skill; it’s delayed feedback. When stakeholders finally see their dashboards, they inevitably say “that’s not quite what I meant” or “we need to slice this differently.” These aren’t communication failures — they’re the natural result of specifying complex analytical requirements through documentation rather than working software. ### AI-Augmented Analytics Delivery [Barton Peveril Sixth Form College](https://barton-peveril.ac.uk/) is a long-established, Ofsted Outstanding post-16 institution serving nearly 5,000 students. In 2024 it became the [UK’s first Google Gemini Academy](https://barton-peveril.ac.uk/blog/sixth-form-college-becomes-the-uks-first-google-gemini-ai-academy/) while undertaking a broader shift toward data-driven decision making. With a director demonstration scheduled for early January and just 19 working days available, Barton Peveril needed a partner who could deliver enterprise-quality analytics at startup speed. ### Early visualisation and design The evening before our first kick-off meeting, we generated ultra-realistic dashboard mockups directly from our statement of work and existing knowledge of higher-education analytics needs. Built using [Loveable](https://lovable.dev/), an AI-powered platform that creates full-stack web applications from natural-language prompts that we’ve integrated into our delivery process via their [API](https://docs.lovable.dev/integrations/lovable-api), these mockups made requirements concrete immediately, letting stakeholders react to something tangible rather than hypothetical. The mockups weren’t throwaway artefacts. They served as design references throughout the build, ensuring the final LookML dashboards matched expectations. By the time we delivered working dashboards, there were no surprises — everyone had already seen and approved the direction. ### Accelerated data modelling and build AI-assisted generation helps us scaffold dbt models, semantic layers and tests far faster than coding by hand. This doesn’t remove the need for good modelling judgment, but it eliminates the mechanical work that historically meant weeks passed before clients saw anything they could provide feedback on. We’ve developed a workflow using Claude Code (soon to-be extended to [Gemini CLI](https://docs.cloud.google.com/gemini/docs/codeassist/gemini-cli)) that starts with requirements documents in a GitHub repo. From there we create dashboard specifications, data catalogs and data models based on [Claude agent Skills](https://code.claude.com/docs/en/skills) that define our working practices, coding standards and required tests. Press enter or click to view image in full size ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F01%2Fdcd17-1arhsfppi178ylvzez8jesa.png%3Fresize%3D700%252C273) Once a consultant reviews and signs off these artefacts, we use them to build the full dbt package structure, from staging through integration to the warehouse layer, complete with internally consistent sample seed data for testing. ### Automated Looker Development & Testing using MCP Toolbox We’ve also developed a Claude Skill defining a complete workflow for AI-assisted Looker development, from initial specification through to validated deployment. It guides Claude Code through a ten-phase process: understanding requirements, examining existing patterns, generating LookML views, explores and dashboards locally, then deploying to Looker via its [Toolbox MCP server](https://rittmananalytics.com/blog/2025/10/13/claude_meets_looker_building_smarter_connected_analytics_with_googles_mcp_toolbox) so it can test every artefact before releasing to production. Press enter or click to view image in full size ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F01%2Fec907-1uvzc5injysn7vzls5k4xda.png%3Fresize%3D700%252C793) The result is a repeatable, testable approach where dashboards are guaranteed to work before being committed, with batch-deployment support for rolling out entire dashboard suites from specification documents. You can read more about our approach on our [AI-Augmented Analytics Delivery services page](https://rittmananalytics.com/services/ai-augmented-delivery), along with more [details on the work we did](https://rittmananalytics.com/case-studies/barton-peveril) for Barton Peveril College. ### What hasn’t changed Several things remain non-negotiable in our delivery approach — and arguably matter more when AI is involved. Every piece of logic we deliver must be explainable by a consultant to a client. When asked why a particular student appears in a cohort count, we can trace the answer through documented business rules in the dbt models. Human review gates still exist before anything goes client-facing. Every dbt model, every LookML view and every dashboard was reviewed by a consultant before deployment. AI accelerates initial versions; humans ensure those versions are correct and appropriate. Semantic design still requires judgment and experience. The decision to implement SCD Type 2 tracking on the student dimension — so the college can analyse how outcomes vary for students whose circumstances change year to year — wasn’t made by AI. It came from conversations with users and their data team about the analytical questions they needed to answer. Business meaning still comes from conversations, not prompts. When we learned that [“Access Plus”](https://barton-peveril.ac.uk/courses/learning-support/) at Barton Peveril means students who are educationally disadvantaged, have SEN or access arrangements, or receive pupil premium or free college meals, that understanding came from stakeholder discussions. Claude Code couldn’t have inferred that definition from the source data. ### Why this changes the client experience The biggest difference clients notice isn’t speed for its own sake — it’s that value appears earlier. Conversations happen around real artefacts rather than abstract documents. Iteration becomes cheaper and safer. More time goes to aligning on meaning. For Barton Peveril, this meant staff saw working dashboards in the first week and started providing feedback immediately through daily interaction with the college’s MIS and data teams. AI-augmented delivery didn’t reduce that collaboration; it made it more productive by ensuring every conversation happened around working software. ## Interested? Find out more! Rittman Analytics is a [boutique data analytics consultancy](https://rittmananalytics.com/about) that helps ambitious, digital-native businesses scale-up their approach to data, analytics and generative AI. We’re authorised delivery partners for Google Cloud along with Oracle, Segment, Cube, Dagster, Preset, dbt Labs and Fivetran and are experts at helping you design an analytics solution that’s right for your organisation’s needs, use-cases and budget and working with you and your data team to successfully implement it. If you’re looking for some help and assistance with your AI initiative or would just like to talk shop and share ideas and thoughts on what’s going on in your organisation and the wider data analytics world, [contact us now](https://calendly.com/markrittman/initial-discovery-call-with-mark-rittman) to organise a 100%-free, no-obligation call — we’d love to hear from you! --- # Rittman Analytics 2025 Wrapped : A Year of Platforms, People and High-Performing Data Teams Canonical: https://rittmananalytics.com/blog/2026/01/19/rittman-analytics-2025-wrapped-a-year-of-platforms-people-and-high-performing-data-teams URL: https://rittmananalytics.com/blog/2026/01/19/rittman-analytics-2025-wrapped-a-year-of-platforms-people-and-high-performing-data-teams Description: Spotify has Wrapped. Everyone else has year end lists. Here’s our take on 2025 at Rittman Analytics. The topics we kept coming back to 1. The semantic layer as the… Author: Mark Rittman Published: 2026-01-19T20:46:52+01:00 Modified: 2026-06-30T19:36:11+01:00 --- ![Rittman Analytics 2025 Wrapped : A Year of Platforms, People and High-Performing Data Teams](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F01%2F0.3619284028022317.webp%3Ffit%3D1400%252C488%26ssl%3D1) Spotify has Wrapped. Everyone else has year end lists. Here’s our take on 2025 at Rittman Analytics. ## The topics we kept coming back to ### 1\. The semantic layer as the foundation for everything else A lot of our writing this year circled around Looker, conversational analytics, and AI assisted BI. The consistent conclusion was that [none of this works without a strong semantic layer underneath](https://rittmananalytics.com/blog/2025/5/11/introducing-conversational-analytics-data-agents-and-code-interpreter-in-looker-and-lookernbspstudio). Without shared definitions, governed metrics, and agreed grain, conversational analytics just accelerates confusion rather than removing it. Press enter or click to view image in full size ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F01%2Fdefae-0dln7um1o6alggshz.png%3Fresize%3D700%252C514) ### 2\. AI that scales judgement rather than replacing it The most interesting AI work we did this year was not about chatbots. It was about packaging judgement, context, and workflow so it can be reused consistently, from [*Deploying Maxwell’s Demon: QA Automation in Droughty*](https://rittmananalytics.com/blog/2025/1/20/deploying-maxwells-demon-qa-automation-in-droughty) through to [*From Prompts to Skills: Automating Financial and KPI Analysis in Looker with Claude Skills and MCP Toolbox*](https://rittmananalytics.com/blog/2025/10/27/from_prompts_to_skills_automating_financial_and_kpi_analysis_in_looker_with_claude_skills_and_mcp_toolbox)*.* In both cases, the goal was the same: take something experienced consultants do well, make it explicit, and then make it repeatable. ### 3\. Data teams versus data functions Our theme mid-year was data team enablement and in [*When Your Talented Data Team Is Stuck in Second Gear*](https://rittmananalytics.com/blog/2025/7/30/when_your_talented_data_team_is_stuck_in_second_gear) our core argument was simple: you can have good people and a modern stack and still fail to deliver value. The limiting factor is usually the operating model, not the talent or the tooling. Press enter or click to view image in full size ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F01%2F79b06-1w0ecgrqykpj44frb5re4jg.png%3Fresize%3D700%252C154) ### 4\. Self service with grown up guardrails Across multiple projects and meetups, self service analytics kept resurfacing as a tension point. Too locked down and teams stall. Too open and trust collapses. The work with clients like [Pleo](https://www.rittmananalytics.com/case-studies/pleo) showed that the answer is not less governance, but shared governance. ## The things that surprised us ### 1\. Project health lives in unstructured data In [*Automating Project RAG Status Reporting with GenAI and Looker*](https://rittmananalytics.com/blog/2025/1/29/genai-looker-rag-status) we found that the most useful delivery signals were hidden in meeting transcripts, Slack messages, and statements of work. Metrics alone rarely tell you whether a project is actually healthy. Press enter or click to view image in full size ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F01%2F0c1bd-0rko8x9e6szjvpucs.png%3Fresize%3D700%252C465) ### 2\. Community and partnerships Still Win We helped kick off the London Looker Meetup and ran two events this year with the support of Google and the wider partner ecosystem. Press enter or click to view image in full size ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F01%2Fcbcb4-0o_o1fslpptnnnh9c.jpg%3Fresize%3D700%252C527) We also started a genuinely exciting collaboration with [Barton Peveril College and Google](https://www.linkedin.com/posts/markrittman_exciting-collaboration-project-im-delighted-activity-7403869337398329344-tc5W?utm_source=share&utm_medium=member_desktop&rcm=ACoAAAAf278B6GXfTUxUU7DUpD-39gZqPf2D5E0) on a BigQuery data lake, Looker dashboards, a central semantic layer, and natural language data access to help improve outcomes in education. Consulting is all about helping people achieve their goals. ### 3\. AI made a small team move very fast Our new website is intentionally treated as a data and AI product, not a brochure. In [*Opinionated, AI Enabled, Modern: The Story Behind the Rittman Analytics Website*](https://rittmananalytics.com/blog/2025/08/29/opinionated-ai-enabled-modern-the-story-behind-rittman-analytics-new-website) we explained how we’d used AI assisted development to move quickly without sacrificing quality. That approach worked better than expected. ### 4\. So is possible to cycle over Ditchling Beacon in September Six of us [rode from London to Brighton for the British Heart Foundation](https://rittmananalytics.com/blog/2025/09/02/rittman-analytics-is-riding-from-london-to-brighton-14th-september-2025-for-the-bhf). This was my third time riding for BHF but the first time the wider team joined in — and the first time I personally made-it over Ditchling Beacon at the end. Press enter or click to view image in full size ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F01%2F2d15a-1i6thmyzkqjjhjcg5ikgq5q.jpeg%3Fresize%3D700%252C394) It was uncomfortable (somewhat of an understatement), rewarding (in the pub afterwards) and a good reminder that work is only one part of what binds a team together. ## Top 5 achievements we are, in a terribly-British way, rather proud of Not like we like to toot-our-own-horn, but this year we’ve delivered more projects, helped enable more data teams and delighted more clients than ever before — thank you to our team and most-importantly, our amazing clients, friends and partners. Press enter or click to view image in full size ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F01%2F7e350-1jwzclci_1h4jvuvfyrtmlg.png%3Fresize%3D700%252C244) 1. **35 projects completed for 23 clients** : Across fintech, retail, SaaS, healthcare, education, and more. Different needs, same outcome: clarity and confidence. 2. **17 new Looker platforms implemented, plus 8 rebooted and relaunched** : A lot of semantic modelling, governance, and practical delivery discipline. Also a lot of “what do we actually mean by revenue” conversations and breathing-into paper bags discreetly. 3. **Revenue up 60% year on year** : Growth, but the good kind: driven by outcomes, not hype. 4. **Consulting team grown by 4** : More capability, more delivery leadership, more people to argue with about naming conventions 5. **Average NPS of 9.47** : We will take that. Happily. And a final note before we wrap-up the year: if 2025 proved anything, it is that organisations do not just need tools. They need partners who can translate ambition into action and complexity into confidence. Modern data done properly is not just a stack, it’s platforms, process, people, and the boring discipline that makes the exciting stuff real. ## Interested? Find Out More! Rittman Analytics is a boutique data analytics consultancy that partners with successful organisations to transform their overwhelmed data teams from a reactive cost centre into a proactive, strategic asset. We deliver more than technology; we deliver the organisational change that makes it work. You can read more about [our approach to modernising data teams and data platforms](https://www.rittmananalytics.com/services/data-team-enablement) on our website, where you can also [take our diagnostic assessment](https://blog.rittmananalytics.com/services/data-team-enablement#diagnostic-assessment) and receive a custom playbook with targeted advice for your specific challenges. Press enter or click to view image in full size ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F01%2Fd7c31-0poqu47dlmubnqbnb.png%3Fresize%3D700%252C174) So if you’re looking for some help and assistance scaling your data capabilities or would just like to talk shop and share your thoughts on what’s going on in your organisation and the wider data analytics world, [contact us now](https://calendly.com/markrittman/initial-discovery-call-with-mark-rittman) to organise a 100%-free, no-obligation call — we’d love to hear from you! --- # You Probably Don’t Need an RFP Canonical: https://rittmananalytics.com/blog/2026/01/19/you-probably-dont-need-an-rfp URL: https://rittmananalytics.com/blog/2026/01/19/you-probably-dont-need-an-rfp Description: Many organisations still use large, prescriptive RFPs to select partners for data platform and analytics projects. These documents often define architecture, scope, timelines and deliverables in detail before any discovery… Author: Mark Rittman Published: 2026-01-19T20:44:16+01:00 Modified: 2026-06-30T19:35:55+01:00 --- ![You Probably Don’t Need an RFP](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F01%2F0.9555669754665426.webp%3Ffit%3D1400%252C764%26ssl%3D1) Many organisations still use large, prescriptive RFPs to select partners for data platform and analytics projects. These documents often define architecture, scope, timelines and deliverables in detail before any discovery work has taken place. Press enter or click to view image in full size ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F01%2F75fd0-1rpndbkwe8ir0cubfoss-pw.png%3Fresize%3D700%252C382) While this approach made sense in a world of static, waterfall-style delivery, it is not well suited to modern data projects where requirements evolve as work progresses and as teams learn from early outputs. Traditional RFP processes rarely lead to the best partner choice. Why? ### 1\. RFPs assume the problem is fully understood upfront Analytics projects frequently involve legacy systems, inconsistent logic, unknown data quality issues and evolving business questions. These cannot be accurately described or costed before a partner has explored the environment. RFPs require partners to act as though all requirements are fixed and well defined. The result is proposals based on assumptions rather than real understanding. Buyers then evaluate partners on how convincing their hypothetical plan sounds, not on how well they collaborate, adapt and solve problems during delivery. Modern data work succeeds when teams can learn, refine and respond as they go. ### 2\. RFP timelines and fixed scopes don’t match the uncertainty of data projects Most RFPs specify detailed milestones, sequencing and deadlines long before the underlying complexity is known. This encourages partners to either overprice to cover risk or underprice to win the work then struggle to deliver within unrealistic constraints. Agile delivery acknowledges uncertainty and manages it through short cycles, working increments and continuous feedback. Fixed scopes undermine this by forcing teams to commit to a plan written without insight into the real environment. ### 3\. RFPs prioritise documentation rather than collaboration A partner’s real value comes from how they think, how they communicate and how they work alongside your team. Those qualities emerge through interaction, not through written submissions. RFPs tend to reward firms that excel at producing polished documents rather than those who excel at designing, building and evolving data platforms. A strong proposal does not guarantee a strong delivery partnership. ### 4\. They create a false sense of where risk sits When a project is framed as a fixed commitment in an RFP, it can appear as if the partner is taking on all risk. In practice, if assumptions turn out to be incorrect, the buyer still carries the consequences through delays, change requests or misaligned solutions. Agile partnerships share risk openly through transparency, iterative delivery and ongoing prioritisation. This creates far healthier outcomes for both sides. ### 5\. RFPs require vendors to guess architecture without real discovery The most effective data architectures emerge from examining real data, existing processes, user needs and organisational constraints. RFPs typically ask partners to define architectural decisions before that insight exists. This leads to solutions that are either overly generic or mismatched to the organisation’s true needs. Discovery must inform design, not the other way around. ### 6\. The selection process becomes transactional when the work requires partnership Successful data projects rely on trust, collaboration and ongoing joint decision-making. RFPs, by nature, create a transactional process where bidders try to minimise perceived risk, present certainty where none exists and optimise for scoring rather than for partnership. This can result in choosing the partner who looks best on paper rather than the one who will deliver the best outcomes. ## A better way to choose the right partner Organisations achieve better results when they replace traditional RFPs with a more collaborative, evidence-based selection process. ### 1\. Start with a small discovery engagement [Run a short engagement](https://www.rittmananalytics.com/services/data-strategy-framework) to explore data sources, prototype initial models, assess feasibility and uncover true risks. This gives both sides real insight into complexity, capabilities and team fit. ### 2\. Use real findings to define a shared roadmap Discovery allows you to build a plan based on evidence rather than assumptions. ### 3\. Deliver through short, value-focused sprints Two-week iterations with clear priorities let teams produce working outputs quickly, gather feedback and adjust direction as needs evolve. ### 4\. Evaluate partners based on how they work, not how they write Look for clarity of thinking, communication, technical judgement, ability to support your team and velocity of delivering real value. ### 5\. Choose partners who can enable long-term ownership The right partner builds platforms that your team can operate and evolve, whether that’s immediately or after a period of structured support. Traditional RFPs are designed for a world of fixed requirements and predictable delivery. Data projects rarely fit that mould. A collaborative selection process grounded in real discovery, agile delivery and early value creation leads to better outcomes for organisations and stronger, more effective partnerships. ## Interested? Find-Out More Rittman Analytics is a boutique data analytics consultancy that partners with successful organisations to transform their overwhelmed data teams from a reactive cost centre into a proactive, strategic asset. We deliver more than technology; we deliver the organisational change that makes it work. You can read more about [our approach to modernising data teams and data platforms](https://www.rittmananalytics.com/services/data-team-enablement) on our website, where you can also take our diagnostic assessment and receive a custom playbook with targeted advice for your specific challenges. So if you’re looking for some help and assistance scaling your data capabilities or would just like to talk shop and share your thoughts on what’s going on in your organisation and the wider data analytics world, [contact us now](https://calendly.com/markrittman/initial-discovery-call-with-mark-rittman) to organise a 100%-free, no-obligation call — we’d love to hear from you! --- # Claude Meets Looker: Building Smarter, Connected Analytics with Google’s MCP Toolbox Canonical: https://rittmananalytics.com/blog/2025/12/08/claude-meets-looker-building-smarter-connected-analytics-with-googles-mcp-toolbox URL: https://rittmananalytics.com/blog/2025/12/08/claude-meets-looker-building-smarter-connected-analytics-with-googles-mcp-toolbox Description: As well as Looker’s new built-in Conversational Analytics feature, you can now connect Looker to AI assistants such as Claude from Anthropic along with agentic developer tools such as Windsurf and Claude… Author: Mark Rittman Published: 2025-12-08T19:51:06+01:00 Modified: 2026-06-30T19:36:06+01:00 --- ![Claude Meets Looker: Building Smarter, Connected Analytics with Google’s MCP Toolbox](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F12%2F0.8426204116330105.webp%3Ffit%3D1400%252C654%26ssl%3D1) As well as Looker’s new built-in [Conversational Analytics](https://blog.rittmananalytics.com/introducing-conversational-analytics-custom-agents-and-code-interpreter-in-looker-and-looker-57cc6d8020d7) feature, you can now connect Looker to AI assistants such as Claude from Anthropic along with agentic developer tools such as Windsurf and Claude Code through [Looker’s new Model Context Protocol (MCP) server](https://cloud.google.com/blog/products/business-intelligence/introducing-looker-mcp-server), available as part of Google’s new [MCP Toolbox](https://googleapis.github.io/genai-toolbox/getting-started/introduction/) open-source toolkit. ### Why Connect Claude to Looker’s Semantic Layer & Tools? While the official Looker Conversational Analytics is a great introduction to natural language querying of data in your Looker semantic layer and is improving release-by-release, connecting Looker as both a trusted source of business data and a set of data visualization tools opens-up a number of new possibilities, including: - **Cross-Explore Analysis:** Go beyond the scope of a single explore to ask complex questions that synthesize data from multiple sources such as Jira, marketing analytics and sales CRMs, all within one conversation. - **External Service Integration:** Connect Looker’s analytical power with other business-critical platforms, enabling you to enrich your data with context from services like Google Drive or Atlassian or even trigger actions, like updating a deal record in HubSpot based on your findings. - **Advanced Agentic Reasoning:** Leverage Claude’s sophisticated reasoning and strategy engine to tackle ambiguous or multi-part questions, resulting in more comprehensive and actionable insights than might otherwise be possible with Looker’s built-in Conversational Analytics feature ## Adding Looker as a Connector to Claude Details on [how to set-up the MCP Server for Looker](https://googleapis.github.io/genai-toolbox/samples/looker/looker_mcp_inspector/) are in the MCP Toolbox docs and get you to the point where you can connect Claude Desktop to Looker via a [local install of MCP Toolbox](https://googleapis.github.io/genai-toolbox/how-to/connect_via_mcp/), or you can deploy MCP Toolbox on the internet via Cloud Run so that it can be accessed by all the users in your Claude Team plan without the need for a local install (\*) Press enter or click to view image in full size ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F12%2Ff973b-1bqfwi3asttfesiojsxbtzw.png%3Fresize%3D700%252C591) If you take a look at the tools this new Looker connector makes available to Claude, as well as querying data from your Looker semantic layer you can also use it to create dashboards and looks to visualize that data, opening-up the possibility of automating whole sections of your data analysis process. Press enter or click to view image in full size ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F12%2F1e259-1r3ddgjlfhicojkesamzqrw.png%3Fresize%3D700%252C591) (\*) *note however the current technical limitation where access to this feature via MCP client, as opposed to Google’s own toolbox SDK, is unauthenticated; this means that you’ll either need to add your own oAuth layer around MCP Toolbox if using it via Cloud Run or restrict usage to Claude Desktop and a locally-running install of MCP Toolbox. Google AgentSpace* [*also uses MCP Toolbox*](https://discuss.google.dev/t/connect-looker-to-agentspace-in-minutes-with-mcp-toolbox-and-the-adk/267516/7) *and Cloud Run but does so using its SDK which is secured, but good-luck getting access to AgentSpace* [*or the connector working*](https://discuss.google.dev/t/connect-looker-to-agentspace-in-minutes-with-mcp-toolbox-and-the-adk/267516/3)*).* ## An Example Claude + Looker Scenario To see how this integration with Claude can work, let’s use a marketing scenario where I’m looking to better-understand the commercial impact of our recent social media blog posts. Using the Claude.ai chat interface I ask the question: Press enter or click to view image in full size ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F12%2F4684d-1jpqehkkglgu1jzlbifolgq.png%3Fresize%3D700%252C328) The thought process under my question shows the thinking Claude’s agentic process applies to answering my question; the preferences the thought process is referring to are ones I’ve added to my Claude profile to help steer it to the main explores I use, and to tell to ignore measures, dimensions and items in the LookML semantic layer marked as “hidden” and not to be used for queries. Press enter or click to view image in full size ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F12%2Fa9295-17mvzdvcj0njrnevr21dszq.png%3Fresize%3D700%252C299) Claude’s response is pretty-comprehensive with a listing of our top landing pages by traffic volume, analysis of our main sources of referral traffic and a set of recommendations on what type of content to write in-future. Press enter or click to view image in full size ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F12%2F878d1-1tz1kxutj64mr2g4yhv6j-g.png%3Fresize%3D700%252C562) Looking at the set of steps Claude ran through to come-up with an answer, you can see the Looker tools from the MCP Server being used used to provide trusted metrics, page view and traffic source data for Claude’s Opus 4.1 LLM. Press enter or click to view image in full size ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F12%2Fb91fd-1h-ehajrzag4a_me1g-p4ka.png%3Fresize%3D700%252C571) ### Leveraging Claude to Answer Complex Looker Questions I then ask a follow-on question, specifically about which blog posts were the most successful in bringing-in traffic that then went-on to look at details of our services, case-studies and other marketing funnel content? Press enter or click to view image in full size ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F12%2F74116-1nfz3_hr-l2cr6vscpcrxug.png%3Fresize%3D700%252C134) Now this is what I call “self-service BI”; with the caveat that, as you’d probably expect from a specialist Looker Google Partner our semantic layer is pretty-well structured and annotated with AI usage in-mind, but Claude is able to understand the intent of our question, find exactly the right items in our Looker explore and come-back with a pretty-insightful answer. Press enter or click to view image in full size ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F12%2F2638d-1fcosjohdqqojzw09wch1tw.png%3Fresize%3D700%252C533) ### Automating the Creation of Looker Dashboards Then I ask Claude to create me a Looker dashboard to monitor the commercial impact of our blog activity, and create documentation for that dashboard in Confluence, another MCP Server we’ve made available to our Claude users. Press enter or click to view image in full size ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F12%2Fc2176-1iunadrkm561enjyy9hbe_q.png%3Fresize%3D700%252C105) Now the dashboard Claude creates isn’t going to win any design awards but it’s a starting point, and the time-saving and completeness it brings to creating user guides in the places users expect to find them is obvious. Press enter or click to view image in full size ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F12%2F12f10-1omqqcgufmgbbiaga5ng2zq.png%3Fresize%3D700%252C565) ### Enriching Your Looker Data using Hubspot and Google Drive And in my final question I make use of Claude’s connectors to our Hubspot CRM system, for richer details on our clients and our historic sales pipeline, along with a connector to our Google Drive documents to add context on our ideal customer profile (ICP): Press enter or click to view image in full size ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F12%2Ff23dd-1fqfm29qrhy0dabdppfiqtq.png%3Fresize%3D700%252C165) Looking at the strategy Claude’s agent uses to answer my question, you can see these new sources along with Looker being used to answer the question: Press enter or click to view image in full size ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F12%2F78bd0-1rsn2ftcgkl-frmvthzutnw.png%3Fresize%3D700%252C504) … and Claude’s final response, incorporating marketing and website data analytics from Looker, CRM data from Hubspot and details of our specific ICP definition from Google Drive is, honestly, better researched and more actionable than most I’ve seen in my career. Press enter or click to view image in full size ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F12%2Fd3abf-1uetui0l5c3rrzssaapzm7q.png%3Fresize%3D700%252C565) ## INTERESTED? FIND OUT MORE! Rittman Analytics is a boutique data analytics consultancy that helps ambitious, digital-native businesses scale-up their approach to data, analytics and [generative AI](https://www.rittmananalytics.com/services/genai). We’re authorised delivery partners for [Google Cloud](https://www.rittmananalytics.com/partners/looker) along with [Oracle](https://www.rittmananalytics.com/partners/oracle), [Segment](https://www.rittmananalytics.com/partners/segment), [Cube](https://www.rittmananalytics.com/partners/cube), [Dagster](https://www.rittmananalytics.com/partners/dagster), [Preset](https://www.rittmananalytics.com/partners/preset), [dbt Labs](https://www.rittmananalytics.com/partners/dbt-labs) and [Fivetran](https://www.rittmananalytics.com/partners/fivetran) and are experts at helping you design an analytics solution that’s right for your organisation’s needs, use-cases and budget and working with you and your data team to successfully implement it. If you’re looking for some help and assistance with connecting Claude to Looker or would just like to talk shop and share ideas and thoughts on what’s going on in your organisation and the wider data analytics world, [contact us now](https://calendly.com/markrittman/initial-discovery-call-with-mark-rittman?utm_source=medium.com&utm_medium=blog&utm_campaign=customer_segmentation) to organise a 100%-free, no-obligation call — we’d love to hear from you! --- # From Prompts to Skills: Automating Financial and KPI Analysis in Looker with Claude Skills and MCP Toolbox Canonical: https://rittmananalytics.com/blog/2025/12/08/from-prompts-to-skills-automating-financial-and-kpi-analysis-in-looker-with-claude-skills-and-mcp-toolbox URL: https://rittmananalytics.com/blog/2025/12/08/from-prompts-to-skills-automating-financial-and-kpi-analysis-in-looker-with-claude-skills-and-mcp-toolbox Description: A couple of weeks ago we shared an article about the new Looker MCP Toolbox and how AI assistants like Claude.ai can now use it to query data from your Looker semantic… Author: Mark Rittman Published: 2025-12-08T19:48:29+01:00 Modified: 2026-06-30T19:35:55+01:00 --- ![From Prompts to Skills: Automating Financial and KPI Analysis in Looker with Claude Skills and MCP Toolbox](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F12%2F0.17482841878719146.webp%3Ffit%3D1400%252C892%26ssl%3D1) A couple of weeks ago we [shared an article](https://blog.rittmananalytics.com/claude-meets-looker-building-smarter-connected-analytics-with-googles-mcp-toolbox-36accd486254) about the new Looker MCP Toolbox and how AI assistants like Claude.ai can now use it to query data from your Looker semantic model, build dashboards, and even manage your Looker instance. However, if you wanted Claude to analyze your Profit & Loss data in Looker using the same kind of structured, multi-step workflow we described in [*How Rittman Analytics Automates our Profit & Loss Reporting and Commentary using VertexAI Gemini 1.5-FLASH and Google BigQuery*](https://rittmananalytics.com/blog/2024/9/17/genai-profit-and-loss-report-analysis), you’d still need to teach Claude your financial analysis process each time. Not only would that be repetitive, but you’d likely hit [Claude’s 200K token limit](https://support.claude.com/en/articles/8606394-how-large-is-the-context-window-on-paid-claude-plans), resulting in the dreaded “This conversation reached its maximum length… Start a new chat” message midway through your analysis. ## Introducing Claude Skills Claude now supports [agent skills](https://support.claude.com/en/articles/12580051-teach-claude-your-way-of-working-using-skills), which improve how it handles specific types of work. A skill is a reusable, filesystem-based resource that gives Claude domain knowledge such as workflows, context, and best practices. In effect, skills turn a general-purpose agent into a specialist. Unlike prompts, which are single-use instructions for one conversation, skills can be loaded when needed. This means you do not have to repeat the same setup or background information each time. We use them to capture: - Our business model and KPI definitions - How to query data sources through the Looker MCP server - Analysis templates and reporting standards - Domain-specific rules and edge cases Skills run in a code execution environment where Claude has access to the filesystem, bash commands, and code execution. Think of them as directories on a virtual machine. Claude moves through them and interacts with files in the same way you would on your own computer. Each skill definition lives in a [SKILL.md](https://support.claude.com/en/articles/12512198-how-to-create-custom-skills) file and the code snippet below shows the first part of the skill we developed to teach Claude financial and performanc analysis as done in our particular organization. \--- name: business-performance-looker description: This skill helps analyze business performance data and generates insights using Looker \--- \# Business Performance Looker Skill This skill provides capabilities for analyzing business performance data and generating insights through Looker integration. \## Capabilities \- \*\*Looker MCP Integration\*\*: Leverage Model Context Protocol (MCP) for seamless Looker connectivity \- \*\*Data Analysis\*\*: Analyze business performance metrics and KPIs from Looker \- \*\*Query Execution\*\*: Run LookML queries and retrieve data from explores and dashboards \- \*\*Report Generation\*\*: Create comprehensive business performance reports \- \*\*Insight Discovery\*\*: Identify trends, patterns, and anomalies in business data \- \*\*Multi-format Export\*\*: Generate Excel, PowerPoint, and PDF reports from Looker data \## How to Use This skill integrates with the Looker MCP server configured in your Claude environment. The MCP server handles authentication and API communication. 1. \*\*Query Data\*\*: Specify the Looker explore, metrics, dimensions, and filters 2. \*\*Analyze Results\*\*: The skill analyzes returned data for insights and patterns 3. \*\*Generate Insights\*\*: Receive actionable business insights based on the data 4. \*\*Create Reports\*\*: Export findings to Excel, PowerPoint, or PDF formats \*\*Note\*\*: Ensure your Looker MCP server is properly configured in Claude's settings with valid credentials. \## Looker Project Structure This skill works with the following Looker configuration: \*\*Model\*\*: Analytics (default) \*\*Explores\*\*: \- \*\*Business Operations\*\* (default) - General business metrics and operations data \- \*\*Finance\*\* - P&L inquiries, financial statements, and accounting data \- \*\*Web Analytics\*\* - Website traffic, page views, user behavior \- \*\*Website Leads\*\* - Sales inquiries, lead generation, conversion tracking \*\*Important\*\*: Always ignore LookML artifacts (dimensions, measures, explores) with the property \`hidden: yes\`. These are internal fields not meant for analysis. \## Input Format Queries can specify: \- Business questions (e.g., "What were Q3 sales?") \- Specific explore to use (defaults to Business Operations) \- Metrics/dimensions to analyze \- Filters and time periods \- Comparison parameters (period-over-period, year-over-year) \## Output Format Results include: \- Data visualizations and charts \- Key performance indicators (KPIs) \- Trend analysis and forecasts \- Actionable business insights \- Formatted reports (Excel, PowerPoint, PDF) \## Example Usage \*\*Business Operations\*\* (default explore): \- "Show me the top 10 products by revenue this month" \- "Create a business performance dashboard with key metrics" \- "What are our operational KPIs for October 2025?" \*\*Finance\*\* (P&L inquiries): \- "Analyze P&L performance for Q3 2025 compared to Q2 2025" \- "Show me gross margin trends over the last 6 months" ... Skills are then packaged-up in .zip files and then uploaded to your Claude account using the Settings page, as shown in the screenshot below. Press enter or click to view image in full size ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F12%2F3ef3b-1ufho2hhdnclpqtiwsdzm2a.png%3Fresize%3D700%252C447) ## Using Claude Code to Build our Looker Claude Skill Of course actually documenting the reporting workflow we use to present our monthly accounts, and the specific way that we analyze our business operations KPIs and where they can be found in our various explores is a non-trivial task that would require someone with deep knowledge of our Looker setup to pull-together. Or … we could take the approach we used in [*Introducing Conversational Analytics, Data Agents and Code Interpreter in Looker and Looker Studio*](https://rittmananalytics.com/blog/2025/5/11/introducing-conversational-analytics-data-agents-and-code-interpreter-in-looker-and-lookernbspstudio) and use the business logic documented in our LookML dashboards along with [Claude Code](https://www.claude.com/product/claude-code) to automate this process. We started with a simple request, starting a new Claude Code session in the root of a git repo we’d created in-order to store the skill definition: "Please review the contents of this repo https://github.com/anthropics/claude-cookbooks/tree/main/skills and get the repo you're working with ready for the creation of a new claude skill" Claude, running within the Claude Code CLI, downloaded [Anthropic’s Claude Cookbook Github repo](https://github.com/anthropics/claude-cookbooks) in order to understand how a Claude skill package should be set-up, then created this structure in a git repo we’d provided for the purpose and then asked us the nature of our business: "We're a professional services consulting firm. Use the Analytics model in Looker, Business Operations explore by default, but Finance explore for P&L queries. Always ignore hidden LookML fields." Claude Code then generated: - a repository structure following Skills best practices - an explore\_router.py utility that automatically routes “P&L” questions to Finance explore - a looker\_config.json with explore-to-use-case mappings - an initial SKILL.md template ready for our business logic ### Inferring our KPIs and business model using Dashboard LookML We then copied-and-pasted the LookML definitions for our financial analytics and monthly business summary dashboards into the Claude Code session: "Here's the LookML for our business\_summary and financial\_analytics dashboards" Claude Code then parsed the LookML within those dashboards and identified our business model: 1. Professional services consulting 2. Revenue recognition separate from invoicing 3. Role-based consultant targets (AE, SAE, PRN) 4. Target ratios: 47% Cost of Delivery, 29% Overheads, 80% Utilization and then created a data dictionary (field\_catalog.json, 412 lines): { "recognized\_project\_revenue.total\_recognized\_revenue\_gbp": { "type": "sum", "description": "Revenue from delivered work, attributed to consultants", "used\_for": "Monthly revenue tracking, consultant attribution", "target\_comparison": "targets.revenue\_target" } } It also inferred from these dashboard LookML definitions: - How each KPI is calculated - RAG thresholds (Red/Amber/Green status indicators) - Role-based targets and hourly rates - Time period handling (MTD, YTD, fiscal year offsets) and generated query templates for - Revenue by consultant - P&L with variance analysis - Pipeline coverage calculations - Utilization trends - 40+ ready-to-use patterns After a few more prompts, Claude Code then built a tailored suite of Python modules for our business model. `**kpi_calculator.py**` **(600 lines)** handled profitability analysis, revenue targets, utilization rates, RAG status, and even generated written summaries of results. `**consultant_analyzer.py**` **(400 lines)** focused on people data—looking up role-based targets, comparing consultant performance, and calculating attribution percentages across teams. `**report_builder.py**` **(500 lines)** automated outputs, creating Excel reports with conditional formatting, PowerPoint slides, and text summaries for quick reviews. As a finishing touch, we asked Claude to include a safeguard: any numbers pulled from the “Revenue and Forecast” explore beyond the current month must be treated as forecasts, never actuals, because mistaking projections for performance could lead to bad business decisions. ## Our Claude Skill for Looker In Action The chat session below, with numbers altered and obfuscated, shows how Claude, guided by this skill and using the Looker MCP Server now gives us the capability of an experienced Rittman Analytics finance department analyst. Press enter or click to view image in full size ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F12%2F5b3c3-1ojbabjwax3av51xsg4pv1w.png%3Fresize%3D700%252C3969) A year ago, we built an impressive GenAI analytics system with LangChain. It worked, but required significant engineering effort and ongoing maintenance. A month ago we discovered [Looker’s MCP Toolbox](https://cloud.google.com/blog/products/business-intelligence/introducing-looker-mcp-server) and eliminated the pre-aggregation bottleneck. Data access became real-time, but each analysis still required prompt engineering. Today, with Claude Skills, we have an AI business analyst that: - Knows our business as well as our senior team - Analyzes data in seconds, not hours - Generates executive-ready reports automatically - Gets smarter with each question asked The future of business intelligence isn’t writing better SQL or building faster dashboards. It’s teaching AI systems to think like your business thinks, analyse like your best analysts analyze, and communicate like your executives communicate … and then getting out of the way and letting them work. ## INTERESTED? FIND OUT MORE! Rittman Analytics is a boutique data analytics consultancy that helps ambitious, digital-native businesses scale-up their approach to data, analytics and [generative AI](https://www.rittmananalytics.com/services/genai). We’re authorised delivery partners for [Google Cloud](https://www.rittmananalytics.com/partners/looker) along with [Oracle](https://www.rittmananalytics.com/partners/oracle), [Segment](https://www.rittmananalytics.com/partners/segment), [Cube](https://www.rittmananalytics.com/partners/cube), [Dagster](https://www.rittmananalytics.com/partners/dagster), [Preset](https://www.rittmananalytics.com/partners/preset), [dbt Labs](https://www.rittmananalytics.com/partners/dbt-labs) and [Fivetran](https://www.rittmananalytics.com/partners/fivetran) and are experts at helping you design an analytics solution that’s right for your organisation’s needs, use-cases and budget and working with you and your data team to successfully implement it. If you’re looking for some help and assistance with your AI initiative or would just like to talk shop and share ideas and thoughts on what’s going on in your organisation and the wider data analytics world, [contact us now](https://calendly.com/markrittman/initial-discovery-call-with-mark-rittman?utm_source=medium.com&utm_medium=blog&utm_campaign=customer_segmentation) to organise a 100%-free, no-obligation call — we’d love to hear from you! --- # An Homage to Oracle Warehouse Builder, 25 Years Ahead of Its Time Canonical: https://rittmananalytics.com/blog/2025/12/08/an-homage-to-oracle-warehouse-builder-25-years-ahead-of-its-time URL: https://rittmananalytics.com/blog/2025/12/08/an-homage-to-oracle-warehouse-builder-25-years-ahead-of-its-time Description: It’s now apparently twenty-five years since the first release of Oracle Warehouse Builder, Oracle’s all-in-one developer tool for building Oracle Database-hosted data warehouses first released back in 2001. Since then the… Author: Mark Rittman Published: 2025-12-08T19:45:01+01:00 Modified: 2026-06-30T19:35:53+01:00 --- ![An Homage to Oracle Warehouse Builder, 25 Years Ahead of Its Time](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F12%2F0.9545894212803949.gif%3Ffit%3D834%252C356%26ssl%3D1) It’s now apparently [twenty-five years since the first release of Oracle Warehouse Builder](https://www.linkedin.com/posts/daveallan_today-i-had-the-chance-to-meet-up-with-a-activity-7402356243467841536-kgqQ), Oracle’s all-in-one developer tool for building Oracle Database-hosted data warehouses first released back in 2001. Since then the world has moved to the cloud, big monolithic ETL suites have decomposed into the modern data stack and everyone’s an analytics engineer working at the command-line using tools such as dbt, git and VSCode. But many a time over these years I’ve looked back at what the original team behind Warehouse Builder managed to achieve all those years ago, and in-particular the [10gR2 release](https://docs.oracle.com/cd/B31080_01/doc/owb.102/b28223/toc.htm) that as a small footnote in its history I had a [small part in helping launch](https://technology.amis.nl/it/odtug-2006-first-report-from-washington/) back in 2006. And so here’s my homage to [Oracle Warehouse Builder](https://en.wikipedia.org/wiki/Oracle_Warehouse_Builder) along with David Allan, Jean-Pierre Dijcks, Paul Narth, Carnot Antonio Romero and everyone else who worked on building these five example features that are still to this day, IMO, twenty-five years ahead of tools in the market today. ## 1\. Rich Model-Driven Data Pipeline Development [dbt Canvas](https://rittmananalytics.com/blog/2024/10/11/coalesce-2024-and-the-launch-of-dbts-visual-editing-experience) and [Coalesce](https://coalesce.io/), for example, now offer graphical mapping editors that provide a visual shell over SQL and a limited set of nodes and predefined SQL patterns. Oracle Warehouse Builder (OWB) took a [much broader and more ambitious approach](https://docs.oracle.com/cd/B31080_01/doc/owb.102/b28223/concept_mappings.htm#CIHGDHFE), providing dozens of genuine transformation operators you could use without writing SQL, covering joins, unions, multilevel aggregators, cached lookups with match rules, pivots, set operators, data quality rules and built-in surrogate key and SCD handling. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F12%2Fbb0cc-0oiip_lvpljnznr_l.gif%3Fresize%3D700%252C299) OWB started with a central metadata model rather than SQL. You defined tables, dimensions, keys, rules and relationships up front and mappings then referenced this model and inherited its semantics. Transformations described intent, not code. Instead of writing SQL, you assembled operators that said what should happen, and OWB’s compiler decided how to implement that logic in the most efficient way for the database. Because the model was authoritative, OWB could auto-generate key handling, merge logic, change detection, audit trails and error management. These were not templates but compiler outputs driven by metadata. Changes to the model cascaded through mappings with consistent validation. Update a datatype or hierarchy and OWB highlighted every affected object and regenerated code where needed, giving Warehouse Builder data pipelines a level of automation, consistency and semantic awareness that most modern ELT tools still do not match ## 2\. Native Semantic Dimensional Modelling OWB treated dimensional modelling as a native part of warehouse design. You didn’t just create tables. [You defined dimensions with surrogate keys, levels, hierarchies, attributes and SCD rules](https://docs.oracle.com/cd/B31080_01/doc/owb.102/b28223/ref_dim_objects.htm#BABEJGDC), and you defined facts with an explicit grain linked to those dimensions. Because OWB understood this model, it could generate surrogate key logic, change detection, hierarchy materialisation and fact-load code without you writing SQL. Change a level or add an attribute and OWB updated dependent mappings automatically. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F12%2Fa1e44-0ffbimapdcnjoypgi.png%3Fresize%3D700%252C652) Modern tools don’t offer this kind of semantic modelling; in dbt a dimension is just another SQL model and [SCDs rely on macros](https://rittmananalytics.substack.com/p/2020-1-3-modelling-slowly-changing-dimensions-type-23-and-6-using-dbt-and-looker). Hierarchies live in documentation rather than in a compiler that knows what they mean. Coalesce improves the experience with templates and metadata but still expands to SQL rather than reasoning about a dimensional model. OWB’s advantage was that it let you declare the structure and behaviour of a warehouse while the system generated and validated the implementation, while most modern tools still put the burden of modelling and correctness back on the engineer. ## 3\. Integrated Data quality and Profiling tightly tied to Mappings OWB let you define data rules, validations, match/merge logic, and quality metrics [directly in the mapping metadata](https://docs.oracle.com/cd/B31080_01/doc/owb.102/b28223/concept_data_quality.htm#CHDFBGJC). These weren’t bolt-on features but part of the design artefacts that affected code generation and audit tables. [OWB’s graphical Data Profiler](https://www.oracle.com/technical-resources/articles/rittman-owb.html) analysed source tables for patterns, domains, outliers and invalid values, letting you spot issues before designing any transformations. From those results you could derive data rules that defined what “good” data looked like, and OWB’s Correction Wizard then generated full correction mappings to fix, standardise or route problem rows. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F12%2F5c0f8-0o7lxsjm0zpg5kqkp.png%3Fresize%3D700%252C545) Because data rules, profiling statistics and correction logic all lived in the same design repository, OWB created a single model that tied data quality directly to mappings rather than treating it as a separate workflow. Press enter or click to view image in full sizToday you often need separate tools: [Monte Carlo](https://www.montecarlodata.com/) for monitoring, [dbt tests](https://docs.getdbt.com/docs/build/data-tests) for schema rules, [Great Expectations](https://greatexpectations.io/) for profiling and quality checks with the result that integration remains piecemeal and not compiler-aware. ## 4\. End-to-end Metadata Lineage Baked into the Platform Because Warehouse Builder was designed to build Oracle data warehouses that often sourced from Oracle applications and presented data to users using Oracle BI tools, it could maintain a unified metadata repository that tracked every object you designed, [giving you full lineage from source column through transformations to targets](https://docs.oracle.com/cd/B31080_01/doc/owb.102/b28223/ref_mdm.htm#CACHGCDG) without extra setup. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F12%2Fe80a8-0kfnaw_g681rbvtse.gif%3Fresize%3D700%252C419) Most modern stacks rely on third-party tools or stitching together metadata from dbt, Fivetran, the warehouse, and BI tools. OWB provided unusually deep lineage and impact analysis because every object in a project lived in a single, strongly typed metadata repository. When you built a mapping OWB recorded not just column-to-column links but the semantic operations applied along the way, giving you the ability to trace any attribute back through every transformation step to its source tables, including lookup paths, surrogate key logic, SCD rules and data-quality corrections. This same engine let you run [impact analysis](https://docs.oracle.com/cd/B31080_01/doc/owb.102/b28223/ref_mdm.htm#CGDJCHAB) in the opposite direction, showing which mappings, downstream tables or reports would be affected if a source column changed. Because this logic operated on the design metadata rather than parsing SQL text, lineage was accurate and consistent even when complex transformations or multi-step expressions were involved. ## 5\. Tight Integration with Oracle’s BI Tools That same metadata model fed directly into Oracle’s BI tools such as Discover and Oracle BI Enterprise Edition. OWB could [publish conformed dimensions, fact tables, hierarchies and even business-friendly item names into Oracle’s BI tools](https://docs.oracle.com/cd/B31080_01/doc/owb.102/b28223/ref_intelligence_objects.htm#g1121756), so analysts saw a curated semantic model rather than raw warehouse structures. Because both tools operated on structured metadata rather than SQL scraping, changes in Warehouse Builder could be propagated into the BI tool semantic layer more reliably, supporting consistent lineage and impact analysis and making those BI tools effectively an extension of the same governed metadata environment. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F12%2F3c96b-0-nboy91wzc3djk4a.gif%3Fresize%3D554%252C385) LookML captures business logic, joins and metrics but it doesn’t inherit transformation semantics from BigQuery or Dataform pipelines, nor do BigQuery or Dataform expose transformation lineage in a way Looker consumes by default. Cube’s [semantic layer sync](https://rittmananalytics.com/blog/2023/11/12/how-rittman-analytics-builds-data-stacks-for-growth-stage-businesses-using-cube-dagster-and-preset) is a step toward tighter integration, but it is still a very different model from the end-to-end metadata flow OWB delivered. Cube can now push parts of its semantic model into downstream tools so measures, dimensions and joins appear natively in environments like Looker, Mode or Hex but it doesn’t provide a single metadata chain from source systems through transformations into analytics. ## So … Why Aren’t We Using Warehouse Builder Today? Which does of course beg the question, if Warehouse Builder was so good, why aren’t we still using it today? Well despite all of those ahead-of-its time features, if you ask anyone who used OWB back in the day and now works with modern, cloud-based data stack tools (i.e. myself and my old friend [Stewart Bryson](https://www.linkedin.com/in/stewartbryson)), we’d also tell you how much of a pain in the a\*\* sometimes OWB was to develop with. - The heavyweight Java thick client meant slow startup, clunky UI interactions and frequent lockups on large projects - Mapping compilation was opaque and could fail deep in the code generator with hard to diagnose errors - Version control was awkward because metadata lived in a central database repository rather than files you could diff or branch; remember git hadn’t even been invented then and with Oracle, everything got stored in an Oracle Database - Deployment across environments relied on control centers and location mappings that often drifted or broke, and extensibility required PL/SQL and custom bindings which made anything beyond the built-ins feel brittle and hard to maintain And in the end, Warehouse Builder fell-out of favour because Oracle-only went out-of-fashion and then Oracle Corp bought [another, almost identical tool](https://www.oracle.com/corporate/pressrelease/oracle-buys-sunopsis-100906.html) that was however more cross-platform (but also more French) and then the two tools got combined into a Frankenstein’s monster that did everything the old tools did but [nobody understood how it worked any more](https://www.rittmanmead.com/blog/2009/09/code-templates-in-owb11gr2-part-1-introduction-and-support-for-heterogeneous-databases/) and [then Oracle killed it](https://danischnider.wordpress.com/2015/07/21/the-end-of-owb-how-to-migrate-existing-owb-projects/). Then the cloud happened, Oracle became the new IBM and all the cool kids rediscovered ELT through Redshift and the open-source dbt (“Data Build Tool”), while us old-timers finally got to do our data warehousing with proper git-based version control, [re-implemented some old OWB features ourselves](https://www.rittmananalytics.com/blog/2020/6/3/column-level-data-profiling-for-google-bigquery-datasets-using-dbt) and pointed out in keynotes when we saw features being released that we first saw in OWB more than twenty years ago. Press enter or click to view image in full size ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F12%2Ffc6a9-0r5b0zqbd5dsul8ft.png%3Fresize%3D700%252C269) ## Interested? Find Out More! Rittman Analytics is a boutique data analytics consultancy that partners with successful organisations to transform their overwhelmed data teams from a reactive cost centre into a proactive, strategic asset. We deliver more than technology; we deliver the organisational change that makes it work. You can read more about [our approach to modernising data teams and data platforms](https://www.rittmananalytics.com/services/data-team-enablement) on our website, where you can also [take our diagnostic assessment](https://blog.rittmananalytics.com/services/data-team-enablement#diagnostic-assessment) and receive a custom playbook with targeted advice for your specific challenges. Press enter or click to view image in full size Press enter or click to view image in full size ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F12%2F7ac7f-0a-wten8myislw8ms.png%3Fresize%3D700%252C174) So if you’re looking for some help and assistance scaling your data capabilities or would just like to talk shop and share your thoughts on what’s going on in your organisation and the wider data analytics world, [contact us now](https://calendly.com/markrittman/initial-discovery-call-with-mark-rittman) to organise a 100%-free, no-obligation call — we’d love to hear from you! --- # “Fivetran In Talks to Buy Dbt Labs”, reports The Information Canonical: https://rittmananalytics.com/blog/2025/09/28/fivetran-in-talks-to-buy-dbt-labs-reports-the-information URL: https://rittmananalytics.com/blog/2025/09/28/fivetran-in-talks-to-buy-dbt-labs-reports-the-information Description: A story in The Information published on Saturday 27th September 2025 reports that “Data Startup Fivetran In Talks to Buy Dbt Labs in Multibillion Dollar Deal”. The article is behind… Author: Mark Rittman Published: 2025-09-28T09:00:00+01:00 Modified: 2026-07-02T00:35:48+01:00 --- ![“Fivetran In Talks to Buy Dbt Labs”, reports The Information](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F1d4e2-1wljnxv_pnyfah3bqzxwkhw.png) A story in The Information published on Saturday 27th September 2025 reports that [“Data Startup Fivetran In Talks to Buy Dbt Labs in Multibillion Dollar Deal”](https://archive.ph/20250928174300/https://www.theinformation.com/articles/data-startup-fivetran-talks-buy-dbt-labs-multibillion-dollar-deal#selection-1187.0-1187.74). ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F1d4e2-1wljnxv_pnyfah3bqzxwkhw.png) The article is behind a paywall but in summary, says that: - Fivetran is in advanced discussions to buy dbt Labs (but is not yet finalized or terms agreed, etc) - The merger aims to create a comprehensive, full-service data platform by combining Fivetran’s data extraction and loading with dbt Labs’ data transformation tools, and designed to help the combined entity better compete against emerging AI-driven competitors. - The acquisition is seen as a strategic step to gain the scale and financial standing (over $400 million in annual recurring revenue) needed for an Initial Public Offering, and could value the new company between $5bn-$10bn *(Disclosure: Rittman Analytics partners with Fivetran, dbt Labs and many other vendors we mention in this blog including Dagster, Google Cloud, Snowflake, Rudderstack, Hightouch and Google Cloud. See* [*our website*](https://www.rittmananalytics.com/) *for more details on partnerships and* [*our previous blogs*](https://www.rittmananalytics.com/blog) *on industry topics)* #### So Why Is This A Big Deal? The modern data stack had at its heart a [best-of-breed, modular philosophy](https://www.getdbt.com/blog/future-of-the-modern-data-stack) with Fivetran synonymous with ingestion (ELT), dbt Labs with transformation and Census with activation. Each layer was designed to interoperate and work to common standards, giving customers the flexibility to swap Fivetran for Airbyte, dbt for SQLMesh or Dataform and Census for Hightouch, for example. If [the story in The Information](https://www.theinformation.com/articles/data-startup-fivetran-talks-buy-dbt-labs-multibillion-dollar-deal) is true and Fivetran, having also [recently acquired reverse-ETL vendor Census](https://www.fivetran.com/press/fivetran-signs-agreement-to-acquire-census-delivering-the-first-end-to-end-data-movement-platform-for-the-ai-era), were to combine these elements into a single organisation it would challenge that model and make the future more like Oracle and Informatica than a loose ecosystem of specialist tools and vendors. And the timing is also quite surprising, given Fivetran only just acquired Tobiko Labs, the team behind SQLMesh and SQLGlot; the combined Christmas Party at least should be fun this year if they put headlines such as [“Is dbt Fusion the death of dbt Core?”](http://Is%20dbt%20Fusion%20the%20death%20of%20dbt%20Core?) and [“dbt + SDF: What Changes and What Doesn’t”](https://www.tobikodata.com/blog/dbt-sdf) in the Christmas crackers and ask people to guess who wrote them. #### What Would This Mean for the Industry? If Fivetran succeeds in bringing ingestion, transformation, and activation into one platform, other modern data stack vendors may need to find partners quickly; reverse ETL, observability, and orchestration are all areas where stand-alone players may increasingly look vunerable and conclude that scale and breadth are necessary to remain relevant. - For example, Hightouch and [RudderStack](https://www.rittmananalytics.com/partners/rudderstack) could look to partner more deeply with observability or orchestration vendors, or even combine with each other, to counterbalance Fivetran + Census. - Prefect and [Dagster](https://www.rittmananalytics.com/partners/dagster) might consider closer alliances with ingestion or transformation players, positioning themselves as the “control plane” for an increasingly consolidated stack. - Monte Carlo, Bigeye, and Soda may need to partner with either warehouses (Snowflake, Databricks) or ingestion vendors to ensure they remain embedded in the flow of data rather than being an add-on. - Cloud data warehouses Snowflake, Databricks and BigQuery have all been building their own ingestion and transformation features (and integrations with dbt such as [dbt Projects on Snowflake](https://docs.snowflake.com/en/user-guide/data-engineering/dbt-projects-on-snowflake); a combined Fivetran–dbt–Census stack would give them a stronger independent competitor. #### What about Customers, Data Teams and dbt’s Community? There are clearly a number of practical advantages that an integrated Fivetran–dbt–Census offering could bring to customers and the broader ecosystem; for example: - Today’s modern data stack often requires careful configuration and monitoring of multiple vendor tools. A single provider for ingestion, transformation, and activation could reduce operational complexity while SQLMesh’s optimisation technology could add smarter scheduling to dbt’s transformation framework. - With multiple vendors, accountability can blur when things go wrong while a vertically integrated provider could offer clearer service levels and a “single throat to choke” for support. But obviously an acquisition of dbt Labs, with its historically open-source roots, could test the balance between community-driven innovation and commercial strategy and might alter perceptions in customers’ minds of neutrality and stewardship. And so even at the level of speculation, there are practical questions for data teams to consider if this deal were to go through: - How much value do you place on modularity versus integration? - What risks are you willing to accept in exchange for simplified vendor management? - How do you protect against lock-in, especially where transformation logic or metadata may become tightly coupled to a single provider? Nothing is confirmed in-terms of this particular deal, but it’s clear that Fivetran is no longer positioning itself as just “the data ingestion company.” The acquisitions of Tobiko Data and Census already showed a move toward platformisation and the possibility of dbt Labs joining that mix highlights the industry trend away from modular ecosystems to integrated platforms. For customers, the right response is not to panic but to plan; now is the time to think carefully about where integration helps, where modularity still matters, and how to preserve optionality in an environment where vendors are consolidating fast. ### INTERESTED? FIND OUT MORE! Rittman Analytics is a boutique data analytics consultancy that helps ambitious, digital-native businesses scale-up their approach to data, analytics and the modern data stack. We’re authorised delivery partners for [Google Cloud](https://www.rittmananalytics.com/partners/looker) along with [Oracle](https://www.rittmananalytics.com/partners/oracle), [Segment](https://www.rittmananalytics.com/partners/segment), [Cube](https://www.rittmananalytics.com/partners/cube), [Dagster](https://www.rittmananalytics.com/partners/dagster), [Preset](https://www.rittmananalytics.com/partners/preset), [dbt Labs](https://www.rittmananalytics.com/partners/dbt-labs) and [Fivetran](https://www.rittmananalytics.com/partners/fivetran) and are experts at helping you and your data team build an analytics solution that’s right for your needs, budget and use-cases. If you’re looking for some help and assistance with your Looker initiative or would just like to talk shop and share ideas and thoughts on what’s going on in your organisation and the wider data analytics world, [contact us now](https://calendly.com/markrittman/initial-discovery-call-with-mark-rittman?utm_source=medium.com&utm_medium=blog&utm_campaign=customer_segmentation) to organise a 100%-free, no-obligation call — we’d love to hear from you! design or scale-up an analytics solution that’s right for your organisation’s needs, use-cases and budget and working with you and your data team to successfully implement it. --- # Google Integrates Spectacles into Looker for Continuous Integration & Regression Testing Canonical: https://rittmananalytics.com/blog/2025/09/08/google-integrates-spectacles-into-looker-for-continuous-integration-regression-testing URL: https://rittmananalytics.com/blog/2025/09/08/google-integrates-spectacles-into-looker-for-continuous-integration-regression-testing Description: When you’re developing LookML views and models in Looker, it’s possible that changes you’ve made could end-up breaking downstream looks and dashboards that rely on that LookML content; you might… Author: Mark Rittman Published: 2025-09-08T21:03:51+01:00 Modified: 2026-06-30T19:35:46+01:00 --- ![Google Integrates Spectacles into Looker for Continuous Integration & Regression Testing](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F09%2F0.21588027490316408.png%3Ffit%3D1400%252C708%26ssl%3D1) When you’re developing LookML views and models in Looker, it’s possible that changes you’ve made could end-up breaking downstream looks and dashboards that rely on that LookML content; you might have deleted or renamed a dimension or measure that’s used in one of your dashboards or you may have a syntax error in your SQL or LookML that prevents a look from returning results. What’s worse, these issues often remain undetected until they end-up impacting your end users, and not-only do they take-up your time in finding and fixing what’s caused the issue, but they undermine the users’ confidence in your system. In the world of software development these are called “regressions”, and regression testing is the process of ensures that previously validated functionality continues to operate as expected after new changes. This type of testing is distinct from [functional testing](https://en.wikipedia.org/wiki/Functional_testing), [performance testing](https://en.wikipedia.org/wiki/Performance_testing) and [user acceptance testing](https://en.wikipedia.org/wiki/Performance_testing) and in the past was performed in a Looker context either by creating [hacky reports or explores](https://michael-scherding.medium.com/looker-tips-how-to-avoid-breaking-changes-560e37b3e38f) that you ran after each release to see if they broke, or you used the [open-source Spectacles tool](https://github.com/spectacles-ci/) or the [hosted and fully-managed version](https://spectacles.dev/) of it [that we covered on this blo](https://www.rittmananalytics.com/blog/2021/05/30/adding-looker-regression-tests-to-the-dbtcloud-ci-cd-pipeline-using-spectacles)g back in 2021. Google have now acquired Spectacles and have integrated its functionality into a new Looker feature called [Continuous Integration](https://cloud.google.com/blog/products/business-intelligence/introducing-continuous-integration-for-looker), with Spectacles’ developers Dylan Baker and Josh Temple now joining the Looker product development team. In this blog co-authored with my colleague Bailey Sharp-Ledger, we’ll explain what Looker Continuous Integration is, how it works and how it compares to the standalone [Spectacles](https://spectacles.dev/products/continuous-integration.html) tool and managed service. Press enter or click to view image in full size ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F09%2F2b4b4-1az0yfbetmsfmpjf5nfed3g.png%3Fresize%3D700%252C276) *Note: As this feature is currently in Public Preview, it’s subject to change and has limited support.* ## What Exactly is Looker CI? At its core, Looker Continuous Integration is about automating the validation of your LookML projects. Think of it as an automated quality assurance (QA) team for your LookML, constantly checking for issues before they affect your users. CI allows you to run a series of comprehensive tests on your LookML code, catching critical errors and inconsistencies across various aspects of your Looker instance. The magic lies in its suite of specialised validators: - SQL Validator: *No more broken fields due to underlying database changes or mistyped SQL*. It runs actual queries against your database, verifying that the dimensions in your Explores exist and execute correctly. You can also configure it to only check specific Explores or exclude certain fields. To avoid consuming resources, queries use `LIMIT 0` and `WHERE 1=2`. - Assert Validator: For those of you leveraging LookML data tests (and you should be!), this validator automatically runs them and flags any failures. This ensures your data transformations & assumptions remain valid. - Content Validator: Ever had a Look or dashboard break because an underlying LookML field was renamed or removed? The Content Validator checks your existing Looks and dashboards for errors, ensuring that all published content remains functional. - LookML Validator: This runs the standard LookML Validator that you’re familiar with from the IDE, checking for syntax errors, referencing issues, & other structural problems within your LookML code itself. These validators are grouped into CI Suites, which are configurable sets of tests associated with your LookML project. You can define *what* it validates, *how* it validates (with specific options), and most importantly, *when* it validates. Press enter or click to view image in full size ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F09%2Ff91e4-1rwkprhxupiypr3pfgpjvxg.png%3Fresize%3D700%252C379) ## Setting Up Your First CI Suite Before you can build CI Suites, a Looker Administrator needs to perform a few one-time setup tasks ### 1\. Admin Prerequisites (Initial Setup) 1. Check compatibility: Ensure your Looker instance is compatible (e.g., Looker-hosted, Public IP for Google Cloud core instances). 2. Enable Continuous Integration: From the Admin panel \> Continuous Integration, underneath the toggle is a warning: *“Don’t enable Continuous Integration on a Looker instance that requires your data to reside in one specific location. Continuous Integration currently stores certain data in the United States.”* 3. Enter CI configuration details: With the enable toggled, an iFrame will appear on the same page. Create a dedicated Continuous Integration user; if this does not appear, check the format of your API Host URL under Admin \> Platform \> API. For example if the url of this page is “[https://rittman.eu.looker.com/admin/api](https://rittman.eu.looker.com/admin/api)”, the API Host URL should be everything before the /admin. 4. Configure Github integration: Optional, but required for Pull Request trigger: install the CI GitHub app in your GitHub organisation. ### 2\. Creating Your First CI Suite Once the admin setup is done, Developers with `manage_ci` permission are able to create a CI suite directly within the Looker IDE. 1. Navigate to the CI Section: In the Looker IDE, under Develop \> your project \> click the new Continuous Integration icon ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F09%2Fc2944-1q7k8k8aruexybopqg_cubq.png%3Fresize%3D678%252C916) 2. Go to Suites: Select the Suites tab. Here, you’ll see any existing CI suites, or you can create a new one. 3. Create a New Suite: Click the Create suite button. 4. Name & Configure: - Suite name: Give your suite a descriptive name (e.g., “Full Project Validation,” “Feature Branch PR Checks”). - Trigger on Pull Requests: This is the essential step for an automated workflow! Toggle “Trigger on pull requests from Looker” to enable automatic CI runs whenever a developer submits a PR to your repository. You can optionally limit this to specific target branches (e.g., only `main` and `develop`). - Enable Validators: Now, select which validators you want to include in this suite. We highly recommend starting by enabling all four: SQL, Assert, Content, and LookML. You can also configure specific options for each validator (e.g., which Explores the SQL validator should test). 5\. Save Your Suite: Click Create suite (or Update suite if editing). 3\. Running Your CI Suite: Your suite will now automatically run on pull requests if configured. However, you can also trigger a run manually: 1. From the Suites page, click the Run suite button next to your configured suite. 2. You’ll be prompted to choose a Workspace: `Production` (to validate your deployed project) or `Dev Mode` (to test a specific development branch), giving the ability to test changes *before* they even get to a PR. Press enter or click to view image in full size ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F09%2Fb4b78-1fnhdcv-whxa7ieifnzqbxg.png%3Fresize%3D700%252C355) ## Comparing Looker CI to Spectacles The big advantage to Looker’s new CI feature compared to the old standalone Spectacles managed service is that it now becomes available to all Looker customers, at no extra cost and built directly into the Looker developer experience. At the time of writing there are a couple of features that were in Spectacles, the managed service, that aren’t in Looker Continuous Integration (but this could change before the feature goes GA) and there are some differences in features that come from the integration into the Looker product. - Spectacles did support integrating with dbt Cloud, so that dbt pull-request test pipelines in Github could then run a Spectacles CI Suite after the dbt build test completed in-order to check that any changes dbt introduced didn’t then break your downstream Looker explores, looks and other content; this feature isn’t in the current release of Looker Continuous Integration but may well be in the GA release later on - Spectacles integrated with GitHub, GitLab, and Azure DevOps to test LookML pull requests automatically and had an API that you could use to build a custom workflow in the CI platform of your choice; Looker has now introduced dedicated CI endpoints in its Looker API that can be used to programmatically start an existing CI Suite run (`[create_ci_run](https://cloud.google.com/looker/docs/reference/looker-api/latest/methods/Project/create_ci_run)`), and check its results (`[get_ci_run](https://cloud.google.com/looker/docs/reference/looker-api/latest/methods/Project/get_ci_run)`). This makes it possible to integrate CI runs directly with orchestration tools such as Apache Airflow, dbt Cloud, or any scheduling system, and embed Looker CI in wider data workflows. - Unlike Spectacles, Looker CI doesn’t include style validators for enforcing LookML coding standards such as consistent naming conventions, proper documentation, or standardised field descriptions; the open-source [Look At Me Sideways (LAMS)](https://github.com/looker-open-source/look-at-me-sideways) is however available though this requires additional setup and isn’t integrated into the CI workflow. - Notification options are limited compared to third-party tools that can integrate with Slack, email, or other communication platforms with customisable messaging; you can however leverage your git platform’s notification system for PR-based CI, or custom build a notification workflow using the Looker API. ## INTERESTED? FIND OUT MORE! Rittman Analytics is a boutique data analytics consultancy that helps ambitious, digital-native businesses scale-up their approach to data, analytics and [generative AI](https://www.rittmananalytics.com/services/genai). We’re authorised delivery partners for [Google Cloud](https://www.rittmananalytics.com/partners/looker) along with [Oracle](https://www.rittmananalytics.com/partners/oracle), [Segment](https://www.rittmananalytics.com/partners/segment), [Cube](https://www.rittmananalytics.com/partners/cube), [Dagster](https://www.rittmananalytics.com/partners/dagster), [Preset](https://www.rittmananalytics.com/partners/preset), [dbt Labs](https://www.rittmananalytics.com/partners/dbt-labs) and [Fivetran](https://www.rittmananalytics.com/partners/fivetran) and are experts at helping you design an analytics solution that’s right for your organisation’s needs, use-cases and budget and working with you and your data team to successfully implement it. If you’re looking for some help and assistance with your Looker initiative or would just like to talk shop and share ideas and thoughts on what’s going on in your organisation and the wider data analytics world, [contact us now](https://calendly.com/markrittman/initial-discovery-call-with-mark-rittman?utm_source=medium.com&utm_medium=blog&utm_campaign=customer_segmentation) to organise a 100%-free, no-obligation call — we’d love to hear from you! --- # How Rittman Analytics Does Financial Analytics using Looker, Fivetran and Google BigQuery Canonical: https://rittmananalytics.com/blog/2025/09/07/how-rittman-analytics-does-financial-analytics-using-looker-fivetran-and-google-bigquery URL: https://rittmananalytics.com/blog/2025/09/07/how-rittman-analytics-does-financial-analytics-using-looker-fivetran-and-google-bigquery Description: As you’d expect from a data analytics consultancy that runs its business on Google Cloud, Google BigQuery and Looker, we use these services to provide the financial reports and dashboards… Author: Mark Rittman Published: 2025-09-07T22:37:02+01:00 Modified: 2025-09-07T22:37:02+01:00 --- ![How Rittman Analytics Does Financial Analytics using Looker, Fivetran and Google BigQuery](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F09%2F0.1669793402786709.png%3Ffit%3D5116%252C3680%26ssl%3D1) # As you’d expect from a data analytics consultancy that runs its business on [Google Cloud, Google BigQuery and Looker](https://www.rittmananalytics.com/partners/googlecloud), we use these services to provide the financial reports and dashboards our finance team needs to understand the trends and direction in our revenue, costs and profitability. In this blog I’ll explain how we source and integrate the data we need from [Xero](https://www.xero.com/uk/), our accounting system and other finance sources, how we’ve built a Looker dashboard for financial analytics and how we’re opening-up this trusted finance data more widely in the company via Google Sheets integration and Looker’s new Conversational Analytics feature. ## Sourcing and Transforming our Xero Finance Data Using [Fivetran](https://www.rittmananalytics.com/partners/fivetran) we extract our accounting data from Xero into Google BigQuery and then transform it, along with timesheet and invoice data and our budgets held in Google Sheets into a finance data mart within our corporate data warehouse. A key part of the logic for these transformations is an adapted version of the [Fivetran Xero dbt package](https://hub.getdbt.com/fivetran/xero/latest/) that generates a `profit_and_loss_report_fact` table that is the source of data for most of our finance reporting each month. Press enter or click to view image in full size Looking at the contents of this [dbt model](https://github.com/fivetran/dbt_xero/blob/main/models/xero__profit_and_loss_report.sql) and table you can see that while its structure suits the needs of a P&L report, if we wanted to create a calculation where, for example, `gross_profit` = `Revenue — Cost of Delivery` we’d have to use quite complicated SQL or Looker table calculations in-order to turn this `net_amount` measure into measures useful for reporting or calculations. Press enter or click to view image in full size So to make this table’s output easier to work with in Looker, we then use LookML [measure filters](https://cloud.google.com/looker/docs/reference/param-field-filters) to create derivations of this `amount` measure for top-level P&L category-level measures such as `Revenue`, `Cost of Delivery`, `Overheads` and `Taxation`. measure: revenue { group\_label: "P&L Actuals" type: sum description: "Sales revenue from consulting projects and other services" value\_format\_name: gbp sql: ${TABLE}.amount;; filters: \[account\_report\_category: "Revenue"\] } In addition we also then use the new [period-over-period measure type](https://cloud.google.com/looker/docs/period-over-period) in LookML to create comparison measures for month-ago, quarter-ago and so on. measure: revenue\_prior\_month { group\_label: "P&L Actuals Prior Month" type: period\_over\_period label: "Revenue Prior Month" description: "Sales Revenue prior month (i.e. previous month)" based\_on: revenue based\_on\_time: period\_month period: month value\_format\_name: gbp sql: ${amount\_base} ;; } Together with our budget data we now have a Looker explore that can be used to provide reporting on the financial results of the company. In the screenshot below the P&L Base Measures folder contains the base-level `Amount` and `Budget` measures we’ll need for creating our P&L report later on, and the derived `Revenue` and other measures will be used when reporting on revenue trends or for comparison or calculations we need to use elsewhere in our dashboard. Press enter or click to view image in full size ## Our Financial Analytics Looker Dashboard The screenshot below shows (with mocked-up numbers) the Financial Analytics dashboard we’re currently using at Rittman Analytics. The key visualisations and reports are highlighted and explained after the screenshot. Press enter or click to view image in full size ### 1\. Headline financial KPIs These are the top-level set of finance KPIs that communicate our YTD performance — “are we currently on-track to hit our revenue number, and are we doing so with a decent net margin?” and are calculated using the `Revenue`, `Cost of Delivery` and other measures we derived from the `net_amount` P&L report measure previously, with additional calculations in the explore queries to calculate % to target as provided by our budgets data source. Press enter or click to view image in full size ### 2\. Variance against budget for our top-level P&L report categories This table-format visualization provides actuals against budget for each top-level (`Revenue`, `Cost of Delivery`etc) P&L report category with the % variance against budget that each actual number represents —answering questions such as *“are our costs increasingly linearly with revenue, at a slower rate than revenue* (which is good) *or at a faster rate* (which is bad)?” Press enter or click to view image in full size ### 3\. Profit & Loss Report This is the report within the dashboard and tells us, by month and YTD, what our `Revenue`, direct (`Cost of Delivery`) and indirect (`Overheads`) costs were along with our provision for corporation tax ( `Taxation`), shareholder distributions ( `Dividends`) and our retained earnings that were left after all these costs (`Total` in the report). Press enter or click to view image in full size P&L reports (or income statements in the US and some other countries) have a particular finance report layout that accountants expect to see, and to create ours we use: - The [**Report Table** custom visualization](https://marketplace.looker.com/marketplace/detail/viz-report_table), which provides additional controls that let’s us specify the column for which we create subtotals in the report, how we display negative values and so on - a **Pivot** of the month dimension in the report together with **Row Totals** and **Totals**, in order to give us one column per month with YTD and retained earnings totals at the bottom of each month column Press enter or click to view image in full size - Sorting of the `account_report_category` and other P&L account dimensions by corresponding `_order`hidden dimensions, so that our report categories such as Revenue, Cost of Delivery, Overheads, and Retained Earnings always appear in the correct order in the report. dimension: account\_report\_category { label: "Account Category" hidden: no order\_by\_field: account\_report\_category\_order type: string description: "Top-level P&L account grouping; Revenue, Cost of Delivery, Overheads, Dividends and Retained Earnings" sql: ${TABLE}.account\_report\_category ;; } ### 4\. Revenue Pipeline & Cumulative Revenue YTD Apart from the detailed P&L report, we also include a few summary visualisations that help us understand trends in our finance numbers. For example, the Revenue Pipeline visualization (as explained in more detail in [this earlier blog](https://rittmananalytics.com/blog/2024/9/23/how-rittman-analytics-analyzes-its-historical-revenue-amp-sales-pipeline-using-google-bigquery-fivetran-stitch-andnbsplooker)) tells me how much of a backlog of agreed and in-negotiation project we had at the end of each month, a good indication of whether we should start hiring or whether in-fact we need to step-up our sales and marketing activity. Press enter or click to view image in full size The cumulative revenue visualization plots this year’s revenue growth curve against previous years, showing me visually whether we’re having a good or bad year compared to the last five. ### **5\. Monthly Average, Min and Max Cash Balance** [*“Revenue is vanity but cash is king”*](https://www.doholis-lambert.com/blog/revenue-is-vanity-profit-is-sanity-but-cash-is-king) goes the old saying in business, and its easy to think that billing = cash in bank whereas it’s often 30, 60 or more days between completing a piece of client work and getting money in the bank, and all the while your team need paying in the meantime. Press enter or click to view image in full size By monitoring the average, highest and lowest balance in our company bank account each month and keeping an eye on when the average might be trending down rather than up, and by understanding the working capital we need to make sure is available in our account each month as bills get paid and money comes in, we can avoid the awkward situation where everyone’s been busy but there’s no money to pay the quarterly VAT bill. ### 6\. Forecast Revenue for the next six months Predicting what our revenue will look-like over the coming months is important as we’ll need to base decisions over hiring, expansion or cutting costs over those numbers. Contracts running over six or twelve months make this task easier but the nature of consulting work is shorter engagements that may not even be in our pipeline at this point. Press enter or click to view image in full size We covered the [forecasting ability within Looker](https://rittmananalytics.com/adding-forecasting-to-your-looker-report-and-dashboards-833670e569d9) on the blog a few years ago and while [this type of forecasting approach](https://cloud.google.com/looker/docs/forecasting-in-visualizations#arima_algorithm) is never going to know about a big contract that ends this month or new ones just about to be signed, it takes a statistical approach that isn’t too far-off the mark as long as you provide it with enough historical data and enable the seasonality feature if your revenue is, well seasonal. For example, the forecast in the screenshot above predicts a fairly-spiky growth in sales when we include two years worth of actual data and ask Looker to forecast forward a further six months, whereas the screenshot below takes three years of data and factors-in seasonality. Press enter or click to view image in full size Which type of forecast is correct for your business will come down to how much your past performance is an indicator of your future, whether your sales are seasonal and so on. In-practice we combine this type of forecast with our schedule of future invoicing from booked work so at least the immediate future is more grounded in facts, not statistical forecasting. ## Google Sheets, Conversational Analytics and Other Routes to Consuming Looker Financials Data The big advantage of putting all your financial analytics data sources and reporting into Looker is the single version of the truth it gives your team; revenue is defined once and consistently across your organisation and you have point-and-click reporting tools that make it easy to find and share insights from your data. ### Looker data as an input into your spreadsheet models But in some cases you’ll have a need to combine this trusted Looker data with spreadsheet models business users’ own and control, and so [Connected Sheets for Looker](https://support.google.com/docs/answer/12822123?hl=en-GB) brings that data, refreshable on-demand or scheduled is a great way to ensure that at least the starting data in a user’s spreadsheet model is correct. Press enter or click to view image in full size ### Providing a Conversational UI for your trusted financial data The appeal of [Looker’s new Conversational Analytics feature](https://www.rittmananalytics.com/blog/2025/5/11/introducing-conversational-analytics-data-agents-and-code-interpreter-in-looker-and-lookernbspstudio) is that it opens-up the trusted data in your Looker semantic model to users who don’t have the time or capability to use Looker’s point-and-click data analysis tools. The challenge though is providing Conversation Analytics with sufficient context on your business and its way of reporting numbers so that it doesn’t just make this up for itself and provide the wrong results back to the user. In my [previous article on this feature](https://www.rittmananalytics.com/blog/2025/5/11/introducing-conversational-analytics-data-agents-and-code-interpreter-in-looker-and-lookernbspstudio) I explained how I used a LookML export of a dashboard to provide this much-needed context, and for our financials data the approach I’ve used is the same: 1. Create a LookML export of the key dashboard for a dataset, in our case the Financial Analytics dashboard we’ve just gone through, and pass it to Gemini with a prompt to turn it into instructions for a Conversational Analytics Agent 2. Review those instructions and fine-tune them with details about the currency to report in, output formats and so on 3. Paste those instructions into the definition of your Conversational Analytics Agent, where you’ll also select the dataset that the agent will use and the persona it will adopt, and 4. Your responses from the Conversational Analytics agent should then be informed by the knowledge and calculation rules that were derived from the LookML version of your financial analytics dashboard Press enter or click to view image in full size ## Interested? Find Out More! Rittman Analytics is a boutique data analytics consultancy that helps ambitious, digital-native businesses scale-up their approach to data, analytics and [generative AI](https://www.rittmananalytics.com/services/genai). We’re authorised delivery partners for [Google Cloud](https://www.rittmananalytics.com/partners/looker) along with [Oracle](https://www.rittmananalytics.com/partners/oracle), [Segment](https://www.rittmananalytics.com/partners/segment), [Cube](https://www.rittmananalytics.com/partners/cube), [Dagster](https://www.rittmananalytics.com/partners/dagster), [Preset](https://www.rittmananalytics.com/partners/preset), [dbt Labs](https://www.rittmananalytics.com/partners/dbt-labs) and [Fivetran](https://www.rittmananalytics.com/partners/fivetran) and are experts at helping you design an analytics solution that’s right for your organisation’s needs, use-cases and budget and working with you and your data team to successfully implement it. If you’re looking for some help and assistance with your AI initiative or would just like to talk shop and share ideas and thoughts on what’s going on in your organisation and the wider data analytics world, [contact us now](https://calendly.com/markrittman/initial-discovery-call-with-mark-rittman?utm_source=medium.com&utm_medium=blog&utm_campaign=customer_segmentation) to organise a 100%-free, no-obligation call — we’d love to hear from you! --- # Rittman Analytics is Riding from London to Brighton, 14th September 2025 for the British Heart Foundation Canonical: https://rittmananalytics.com/blog/2025/09/02/rittman-analytics-is-riding-from-london-to-brighton-14th-september-2025-for-the-british-heart-foundation URL: https://rittmananalytics.com/blog/2025/09/02/rittman-analytics-is-riding-from-london-to-brighton-14th-september-2025-for-the-british-heart-foundation Description: Six of the Rittman Analytics team are swapping SQL for saddles and Looker for Lycra as we cycle from London to Brighton on Sunday, 14th September 2025 in support of… Author: Mark Rittman Published: 2025-09-02T22:13:41+01:00 Modified: 2025-09-02T22:13:41+01:00 --- ![Rittman Analytics is Riding from London to Brighton, 14th September 2025 for the British Heart Foundation](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F09%2F0.2040330260704829.jpeg%3Ffit%3D1344%252C768%26ssl%3D1) # Six of the Rittman Analytics team are swapping SQL for saddles and Looker for Lycra as we cycle from London to Brighton on Sunday, 14th September 2025 in support of the British Heart Foundation. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2F0.7454360782250545.jpeg%3Fssl%3D1) We’re proud to be pedalling for a purpose. Every mile we ride is to help fund life‑saving research into heart and circulatory diseases. The BHF has already helped halve death rates from these conditions in the UK, but there’s more to be done. Every day, hundreds of lives are still lost. With your support, we can help change that. Just £24 could fund two hours of research by an early‑career scientist. Every pound helps whether it powers a lab or gets us up the next hill! # The Ride On Sunday, 14 September 2025, we’ll take on the iconic London to Brighton Cycle Ride: - Starting from Clapham Common, London - Covering 55 miles through the Surrey and Sussex countryside - Tackling the infamous Ditchling Beacon climb - And finally crossing the finish line on the Brighton seafront ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2F0.4599499448903035.png%3Fssl%3D1) # How you can help Please donate whatever you can to help us hit our target. [Donate to our British Heart Foundation JustGiving page →](https://www.justgiving.com/page/rittman-analytics-l2b-2025) If your company offers donation matching, this is a great time to use it. # Why BHF? Heart and circulatory diseases affect millions of people and families across the UK. The BHF funds the researchers, breakthroughs, and front‑line initiatives that turn the tide from prevention through to treatment and recovery. This will be the third time Mark Rittman has completed the London to Brighton ride to raise money for the BHF but the first time the whole team has joined him, so if you’re in Brighton on the day, come say hello at the seafront – we’ll be the data folk in lycra..! --- # Opinionated, AI-Enabled & Modern: The Story Behind Rittman Analytics’ New Website Canonical: https://rittmananalytics.com/blog/2025/08/29/opinionated-ai-enabled-modern-the-story-behind-rittman-analytics-new-website URL: https://rittmananalytics.com/blog/2025/08/29/opinionated-ai-enabled-modern-the-story-behind-rittman-analytics-new-website Description: Over the past few years, we’ve built and modernised data platforms for dozens of digitally native businesses. The patterns are consistent: ambitious data goals, strong technical leadership and newly formed… Author: Mark Rittman Published: 2025-08-29T17:42:42+01:00 Modified: 2025-08-29T17:42:42+01:00 --- ![Opinionated, AI-Enabled & Modern: The Story Behind Rittman Analytics’ New Website](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F08%2F0.8472241509609719.png%3Ffit%3D3840%252C2785%26ssl%3D1) # Over the past few years, we’ve built and modernised data platforms for dozens of digitally native businesses. The patterns are consistent: ambitious data goals, strong technical leadership and newly formed teams still finding their way. Our new website at [https://www.rittmananalytics.com](https://www.rittmananalytics.com/) reflects those realities, with messaging and services tightly aligned to the challenges our clients actually face, leading on: 1. [**Data Platform Implementation**](https://www.rittmananalytics.com/services/data-platform-development) — building modern, scalable, reliable analytics platforms on dbt, BigQuery/Snowflake, Looker, and Cube. 2. [**Data Team Enablement**](https://www.rittmananalytics.com/services/data-team-enablement) — transferring knowledge, building confidence, and embedding best practices so teams come out stronger, not dependent. Press enter or click to view image in full size [Case studies](https://www.rittmananalytics.com/case-studies) feature front-and-centre on the new site because they show, in our clients’ own contexts, how ambitious data goals and modern stack challenges were solved through our [Data Platform Implementation](https://www.rittmananalytics.com/services/data-platform-development) and [Data Team Enablement](https://www.rittmananalytics.com/services/data-team-enablement) services, making the value we deliver concrete, credible and immediately relatable to readers. Press enter or click to view image in full size ## Making Blogs and Podcast Episodes Discoverable Our [blog](https://www.rittmananalytics.com/blog) and [Drill to Detail Podcast](https://www.rittmananalytics.com/drilltodetail) have always been the most popular part of the site, but even we found it hard to discover past content — so we’ve added filtering, grouping, and sorting by tag or service category and now give you AI-powered recommendations on other posts you’d find interesting. Press enter or click to view image in full size We’ve also made it easier to discover the services we offer and the partner technologies we specialise in delivering, with featured services, partners and industry verticals to highlight content we think you’ll be interested in. Press enter or click to view image in full size ## Built and Designed Around Data, Analytics & AI When we help clients modernise their data platforms, we always stress the importance of building with the right tools and showing what good looks like. So when it came to redesigning our own site, we applied the same thinking. This isn’t just a marketing website; it’s a working showcase of the technologies and solutions we champion every day. Press enter or click to view image in full size - **Front-end**: Built with React, Vite, TypeScript, Tailwind, and deployed on Netlify for a fast, modern user experience. - **Backend**: Running on Supabase, with Edge Functions providing clean integrations to external services such as Open AI and HubSpot - **Analytics**: We’ve instrumented the site with Segment, store event data in BigQuery, and bring it back into the site through embedded Looker dashboards Press enter or click to view image in full size - **AI Features**: We’ve integrated OpenAI’s GPT-4o via the API and Supabase Edge Functions to power features such as [personalised Data Team Enablement playbooks](https://www.rittmananalytics.com/services/data-team-enablement), Blog post recommendations for easier discovery, Autotagging to keep content structured and for our blog authors, Authoring and style-guide tools for consistent content quality Press enter or click to view image in full size - **Development**: Built with the support of [Loveable.dev](https://lovable.dev/) and [Claude Code](https://www.anthropic.com/claude-code), using modern AI-assisted workflows that speed up iteration without sacrificing code-quality. We chose to develop the site using the same AI-assisted workflows we’re helping clients adopt. **Loveable.dev** gave us the scaffolding and rapid prototyping environment to spin up React components and iterate quickly without the drag of boilerplate setup. For the more complex, site-wide features, we leaned on Claude Code as a development partner. It helped us design and implement event tracking with Segment across the site, and build the logic for indexing content into vector tables in Supabase — the foundation for our RAG semantic search and other AI-powered features. In practice, Claude Code allowed us to move faster, reduce context-switching, and maintain consistency in TypeScript, React, and Supabase code across the entire platform. By combining these tools with our own stack, we’ve turned the website into more than a shop window: it’s a **live example of how AI can accelerate development and make ambitious features achievable for small, focused teams**. Press enter or click to view image in full size ## Interested? Find Out More! Rittman Analytics is a boutique data analytics consultancy that partners with successful organisations to help them innovate faster and automate their analytical processes. We deliver more than technology; we also deliver the organisational change that makes it work. You can read more about [our approach to modernising data teams and data platforms](https://www.rittmananalytics.com/services/data-team-enablement) on our website, where you can also [take our diagnostic assessment](https://medium.com/services/data-team-enablement#diagnostic-assessment) and receive a custom playbook with targeted advice for your specific challenges. So if you’re looking for some help and assistance scaling your data capabilities or would just like to talk shop and share your thoughts on what’s going on in your organisation and the wider data analytics world, [contact us now](https://calendly.com/markrittman/initial-discovery-call-with-mark-rittman) to organise a 100%-free, no-obligation call — we’d love to hear from you! --- # Adventures in Aggregate Awareness (and Level-Specific Measures) with Looker Canonical: https://rittmananalytics.com/blog/2025/08/25/adventures-in-aggregate-awareness-and-level-specific-measures-with-looker URL: https://rittmananalytics.com/blog/2025/08/25/adventures-in-aggregate-awareness-and-level-specific-measures-with-looker Description: Looker aggregate awareness is a performance optimisation feature that automatically directs users’ queries to use pre-aggregated summary tables instead of detail-level transaction tables, significantly improving the performance of reports and… Author: Mark Rittman Published: 2025-08-25T22:02:38+01:00 Modified: 2025-08-25T22:02:38+01:00 --- ![Adventures in Aggregate Awareness (and Level-Specific Measures) with Looker](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F08%2F0.01426986606877012.png%3Ffit%3D2048%252C676%26ssl%3D1) # Looker [aggregate awareness](https://cloud.google.com/looker/docs/aggregate_awareness#looker_unions_new_data_to_your_aggregate_tables) is a performance optimisation feature that automatically directs users’ queries to use pre-aggregated summary tables instead of detail-level transaction tables, significantly improving the performance of reports and dashboards that return aggregated data. Once you have enabled aggregate awareness in your LookML model, Looker intelligently works-out when a query can be satisfied by a smaller, pre-computed [aggregate table](https://cloud.google.com/looker/docs/reference/param-explore-aggregate-table) that contains the necessary dimensions and measures at the appropriate granularity, and then automatically rewrites the SQL so that it queries the summary table instead of the larger detail-level table. Press enter or click to view image in full size This happens transparently to end users who continue to explore data normally while benefiting from faster query times, and the semantic layer you set-up tells Looker that daily roll-up can serve weekly/monthly/yearly queries, monthly can serve quarterly/yearly and so on. # Enabling Aggregate Awareness in Looker Enabling this feature within your LookML model starts with defining aggregate tables within an explore using the `aggregate_table` parameter. Each [aggregate table](https://cloud.google.com/looker/docs/reference/param-explore-aggregate-table) specification includes a query definition with dimensions and measures, together with a [materialization](https://cloud.google.com/looker/docs/reference/param-explore-aggregate-table#materialization) configuration that controls how the aggregate table is persisted. In the LookML example below, this explore defines two aggregate tables on top of the `orders` explore to improve query performance: - one rolls up sales at the daily store–category level with measures like revenue, order count, and average order value, - while the other summarizes sales at the monthly category–customer segment level (restricted to completed orders) with measures including unique customers. - Each aggregate table has its own refresh logic (`datagroup_trigger` for the daily table, `sql_trigger_value` with partitioning for the monthly table), and Looker will automatically use them when a query’s selected fields and filters match their definitions, falling back to the base `orders` table and its joins to `products` and `stores` when needed. explore: orders { aggregate\_table: daily\_store\_sales { materialization: { datagroup\_trigger: daily\_etl\_datagroup } query: { dimensions: \[orders.order\_date, stores.store\_id, products.category\] measures: \[orders.total\_revenue, orders.order\_count, orders.average\_order\_value\] timezone: America/Los\_Angeles } } aggregate\_table: monthly\_category\_sales { materialization: { sql\_trigger\_value: SELECT MAX(updated\_at) FROM orders ;; partition\_keys: \["order\_month"\] # ensure this matches the aggregate column name } query: { dimensions: \[orders.order\_month, products.category, customers.customer\_segment\] measures: \[orders.total\_revenue, orders.order\_count, orders.unique\_customers\] filters: \[orders.status: "completed"\] } } join: products { sql\_on: ${orders.product\_id} = ${products.id} ;; } join: stores { sql\_on: ${orders.store\_id} = ${stores.id} ;; } } So how does Looker’s aggregate awareness feature work? #### Aggregate Navigation Powered by Looker’s Semantic Model Conceptually, Looker’s aggregate awareness works by maintaining a mapping between detailed base explores and their pre-aggregated counterparts through aggregate table definitions in your LookML model. When a user runs a query, Looker’s query engine examines the requested dimensions and measures, then checks if there’s a matching aggregate table that contains all the required fields at an appropriate granularity. If found, it automatically rewrites the generated SQL to query the smaller aggregate table instead of the base table. Press enter or click to view image in full size The system tracks which dimensions can be “rolled up” (like date to month, or product to category) and which measures are additive, allowing it to determine when an aggregate table is a valid substitute for querying the base-level table. Behind the scenes, Looker maintains aggregate tables as [persistent derived tables](https://cloud.google.com/looker/docs/derived-tables#persistent_derived_table) (PDTs) for which it takes responsibility for refreshing as specified by their materialisation settings. It then acts as an “intelligent router” that transparently redirects queries to the optimal table based on the query pattern, without the end user needing to know multiple aggregations of the base-level table exist or manually choose between detailed and summary versions. ## Incrementally-Refreshing your Aggregates As is the case with PDTs, you should generally try to [refresh your aggregate awareness tables incrementally](https://cloud.google.com/looker/docs/aggregate_awareness#aggregate_tables_must_be_persisted) (rather than as full rebuilds) to incorporate new and changed data as it arrives in the base-level transaction table. Incremental refresh for aggregate tables in Looker works by rebuilding only recent time periods of an aggregate table rather than recreating the entire table from scratch. When you define an `increment_key` (a date/time dimension) and an `increment_offset` (number of periods to rebuild), Looker automatically deletes data for those recent periods from the existing aggregate table and rebuilds just those periods from the source data during each refresh, such as in the LookML example below. \# Daily aggregate with incremental refresh aggregate\_table: daily\_store\_sales { query: { dimensions: \[order\_date, stores.store\_id, products.category\] measures: \[total\_revenue, order\_count, average\_order\_value\] timezone: "America/Los\_Angeles" } materialization: { datagroup\_trigger: daily\_etl\_datagroup increment\_key: "order\_date" increment\_offset: 3 # Rebuild last 3 days for late-arriving data } } In addition, when an aggregate table is built with a time filter [Looker can UNION fresh data to the aggregate table](https://cloud.google.com/looker/docs/aggregate_awareness#looker_unions_new_data_to_your_aggregate_tables) to keep results up to date. For example, suppose you maintain an aggregate table covering the last three days, but it was last rebuilt yesterday; on its own, that table wouldn’t include today’s records so you wouldn’t normally expect it to serve a query asking for the most recent daily data. In this case, Looker compensates by querying today’s data directly and then unioning those fresh results with the aggregate table. This behaviour applies when the aggregate table: - has a time filter defined, and - includes a dimension derived from the same time field as that filter. #### Looker-Managed vs. Database-Managed Aggregates Of course data warehousing platforms such as Google BigQuery and Snowflake data warehouse have their own, [in-built aggregate management capabilities](https://cloud.google.com/bigquery/docs/materialized-views-intro) which typically have more features and cover a wider range of queries and aggregation functions than just the `SUM`, `COUNT`, `AVERAGE`, `MIN`, and `MAX` that Looker permits. As a rule-of-thumb, you should choose Looker’s aggregate awareness feature when you need to use business logic in the aggregate definition that only exists in your LookML model, or when your team lacks lacks database admin privileges but has Looker development access; either way the resulting aggregate tables still go in your data warehouse but it’s Looker creating them, not you using SQL scripts or your database admin console. In most other cases when you can have your warehouse define and maintain those aggregates and take-care of query redirection for you in the background, this generally is more efficient solution with a wider set of use-cases it can cover and it keeps this complexity out of your Looker model, just as using dbt or Dataform for data transforming makes more sense than trying to do all of this via LookML. If this is as complex as your aggregate awareness needs are likely to get, you can stop reading now and everthing’s covered. If however you’re one of those complicated enterprise customers, typically in the retail vertical where you’re migrating all of this from Oracle and some legacy old BI tool along with an ETL process that already creates and manages your summary tables, go and get yourself a stiff drink and read on. ### The Challenge in Using Externally-Created Aggregate Tables There are, of course, situations where you want to create and manage your aggregate tables external to Looker, but you still want Looker to make use of those tables when running dashboards and reports that require aggregated data in their results. For example, a client we helped implement Looker for recently had already invested a substantial amount of time creating ETL routines to build and manage their BigQuery summary tables and expected to be able to configure Looker’s aggregate awareness to use those, rather then build them all again itself. Unfortunately however, their assumption was incorrect as Looker’s aggregate awareness can only use aggregate tables that it actually creates and manages. Why? Because Looker needs specific metadata to make aggregate awareness (specifically, query rewriting) work and it can’t get that information from database objects that you’ve created independently of Looker. #### Manually Configuring Aggregate Awareness using Liquid and \_in\_query If for whatever reason recreating your aggregate table logic and refresh routines in Looker using aggregate awareness just isn’t practical, then there is a [hybrid approach that was used before this feature became available in Looker](https://discuss.google.dev/t/advanced-lookml-intro-aggregate-awareness/121151) that you can use, with caveat that it’s more complex and fragile to implement than just using built-in aggregate awareness. This hybrid approach leverages the [Liquid templating language](https://cloud.google.com/looker/docs/liquid-variable-reference) you can embed in your LookML model along with the [`_in_query` variable](https://cloud.google.com/looker/docs/liquid-variable-reference#in_query) that tells you whether certain fields such as a daily, monthly, or yearly timeframe field are present in the query. Hang on to your hats though as it gets complex, quickly. In the example code below we’ve creates a single “router” LookML view with a `derived_table` that picks the right physical table via Liquid, along with measures that switch between summing pre-aggregated values vs. computing using aggregations of your detail-level data. view: orders\_base { sql\_table\_name: schema.orders ;; } view: daily\_store\_sales { sql\_table\_name: schema.daily\_store\_sales ;; # (order\_date, store\_id, category, total\_revenue, order\_count, average\_order\_value, …) } view: monthly\_category\_sales { sql\_table\_name: schema.monthly\_category\_sales ;; # (order\_month, category, customer\_segment, total\_revenue, order\_count, unique\_customers, …) } \# 2) Routing view that standardizes the schema and chooses the source with Liquid view: orders\_router { derived\_table: { # IMPORTANT: align/alias columns so the same field names exist no matter which table we pick. # Add/remove columns to match your schema. sql: SELECT -- time CAST(order\_date AS DATE) AS order\_date, CAST(order\_month AS DATE) AS order\_month, -- dims store\_id, category, customer\_segment, status, product\_id, -- pre-agg measures (present in agg tables; null in base when not applicable) total\_revenue, order\_count, average\_order\_value, unique\_customers, -- raw columns (present in base; null in aggs) revenue, user\_id FROM {% if orders\_router.order\_date.\_in\_query or stores.store\_id.\_in\_query or products.category.\_in\_query %} ${daily\_store\_sales.SQL\_TABLE\_NAME} {% elsif orders\_router.order\_month.\_in\_query or customers.customer\_segment.\_in\_query %} ${monthly\_category\_sales.SQL\_TABLE\_NAME} {% else %} ${orders\_base.SQL\_TABLE\_NAME} {% endif %} ;; } # ------- Dimensions (shared names across all sources) ------- dimension\_group: order\_date { type: time timeframes: \[date, week, month, year\] sql: ${TABLE}.order\_date ;; } # Optional explicit month dim if you use it directly in Explores dimension: order\_month { type: date sql: ${TABLE}.order\_month ;; } dimension: store\_id { sql: ${TABLE}.store\_id ;; } dimension: category { sql: ${TABLE}.category ;; } dimension: customer\_segment { sql: ${TABLE}.customer\_segment ;; } dimension: status { sql: ${TABLE}.status ;; } dimension: product\_id { sql: ${TABLE}.product\_id ;; } dimension: user\_id { hidden: yes sql: ${TABLE}.user\_id ;; } # only populated when base table is used # ------- Measures with Liquid fallbacks ------- measure: total\_revenue { type: number sql: {% if orders\_router.order\_date.\_in\_query or stores.store\_id.\_in\_query or products.category.\_in\_query or orders\_router.order\_month.\_in\_query or customers.customer\_segment.\_in\_query %} -- On aggregate tables, sum the pre-aggregated value SUM(${TABLE}.total\_revenue) {% else %} -- On base table, compute from raw SUM(${TABLE}.revenue) {% endif %} ;; } measure: order\_count { type: number sql: {% if orders\_router.order\_date.\_in\_query or stores.store\_id.\_in\_query or products.category.\_in\_query or orders\_router.order\_month.\_in\_query or customers.customer\_segment.\_in\_query %} SUM(${TABLE}.order\_count) {% else %} COUNT(\*) -- or COUNT\_DISTINCT order\_id if appropriate {% endif %} ;; } measure: average\_order\_value { type: number value\_format: "£#,##0.00" sql: {% if orders\_router.order\_date.\_in\_query or stores.store\_id.\_in\_query or products.category.\_in\_query %} -- Prefer pre-aggregated AOV if provided AVG(${TABLE}.average\_order\_value) {% elsif orders\_router.order\_month.\_in\_query or customers.customer\_segment.\_in\_query %} AVG(${TABLE}.average\_order\_value) {% else %} NULLIF(SUM(${TABLE}.revenue),0) / NULLIF(COUNT(\*),0) {% endif %} ;; } measure: unique\_customers { type: number sql: {% if orders\_router.order\_month.\_in\_query or customers.customer\_segment.\_in\_query %} SUM(${TABLE}.unique\_customers) {% else %} COUNT\_DISTINCT(${TABLE}.user\_id) {% endif %} ;; } } \# 3) Explore uses the router as its FROM and keeps your joins the same explore: orders { from: orders\_router join: products { relationship: many\_to\_one sql\_on: ${orders\_router.product\_id} = ${products.id} ;; } join: stores { relationship: many\_to\_one sql\_on: ${orders\_router.store\_id} = ${stores.id} ;; } join: customers { relationship: many\_to\_one sql\_on: ${orders\_router.user\_id} = ${customers.id} ;; # Note: user\_id only exists when falling back to base; for agg tables, unique\_customers is pre-agg. } } - If a user selects daily fields or store/category detail, the router LookML view pulls from `daily_store_sales` whereas if they pivot by `order_month` or slice by `customer_segment`, it pulls from `monthly_category_sales` , otherwise it falls back to `orders_base` - Measures switch between summing pre-aggregated columns (on aggs) and calculating from raw (on base), so users keep one explore and get the fastest table automatically. ## Level-Specific Measures and Aggregate Awareness If it were just the effort involved in recreating their existing aggregate management logic in LookML aggregate tables, I’d advise the client to just suck it up and rebuild it in Looker if they wanted the certainty of managing aggregate management and navigation in Looker But more significantly they also needed all of this to support a number of measures that existed only at specific aggregation levels, for example, promotional discounts tracked at category level and shrinkage recorded at department level, as shown in the diagram below. Press enter or click to view image in full size The problem with level-specific measures such as the promotional data and inventory-shrinkage data in this example, is that Looker’s aggregate awareness feature: - requires all measures to exist in every aggregate table (or be computable from existing columns) - can’t handle measures that are NULL at certain aggregation levels - would fail when trying to route queries for measures that don’t exist in certain tables To solve this problem we helped them implement an extension of the hybrid aggregate awareness solution that: - Only queries tables that actually contain the requested measures - Explicitly returns NULL rather than incorrect values when measures are incompatible with the requested grain - Maintains a single explore interface while transparently handling the complexity view: sales\_router { derived\_table: { sql: SELECT -- Keys at both grains (NULL where not applicable) department\_id, category\_id, -- Measures at their native grains promo\_discount, -- only populated by category-level PDT shrinkage, -- only populated by department-level PDT revenue, order\_id, user\_id FROM {% if sales\_router.promo\_discount.\_in\_query or products.category.\_in\_query %} ${category\_agg.SQL\_TABLE\_NAME} -- category grain PDT (has promo\_discount) {% elsif sales\_router.shrinkage.\_in\_query or products.department.\_in\_query %} ${department\_agg.SQL\_TABLE\_NAME} -- department grain PDT (has shrinkage) {% else %} ${orders\_base.SQL\_TABLE\_NAME} -- raw fallback {% endif %} ;; } dimension: department\_id { sql: ${TABLE}.department\_id ;; } dimension: category\_id { sql: ${TABLE}.category\_id ;; } # Category-grain measure measure: promo\_discount { type: number sql: {% if products.category.\_in\_query or sales\_router.promo\_discount.\_in\_query %} SUM(${TABLE}.promo\_discount) {% elsif products.department.\_in\_query or sales\_router.shrinkage.\_in\_query %} NULL -- incompatible grain: category measure asked for at department grain {% else %} -- If neither grain is explicitly asked, allow safe roll-up from category PDT SUM(${TABLE}.promo\_discount) {% endif %} ;; } # Department-grain measure measure: shrinkage { type: number sql: {% if products.department.\_in\_query or sales\_router.shrinkage.\_in\_query %} SUM(${TABLE}.shrinkage) {% elsif products.category.\_in\_query or sales\_router.promo\_discount.\_in\_query %} NULL -- incompatible grain: department measure at category grain {% else %} -- If grain not explicit, safe roll-up from department PDT SUM(${TABLE}.shrinkage) {% endif %} ;; } # Generic measures that work at any grain (compute from base when needed) measure: total\_revenue { type: number sql: {% if products.category.\_in\_query or sales\_router.promo\_discount.\_in\_query %} SUM(${TABLE}.revenue) -- category PDT has revenue at category grain {% elsif products.department.\_in\_query or sales\_router.shrinkage.\_in\_query %} SUM(${TABLE}.revenue) -- department PDT has revenue at department grain {% else %} SUM(${TABLE}.revenue) -- base fallback {% endif %} ;; } } This manual aggregate awareness routing system solves the level-specific measure problem through two key mechanisms: **1\. Dynamic Table Selection** : The derived table uses Liquid templating to choose the appropriate source table based on what’s in the query: - If `promo_discount` or `category` is requested → routes to `category_agg` table - If `shrinkage` or `department` is requested → routes to `department_agg` table - Otherwise → falls back to `orders_base` table **2\. Measure-Level NULL Handling** : Each measure contains logic to return NULL when requested at an incompatible grain: For `promo_discount` (only exists at category level): - Returns `SUM(promo_discount)` when category grain is requested - Returns `NULL` when department grain is requested (preventing incorrect aggregation) For `shrinkage` (only exists at department level): - Returns `SUM(shrinkage)` when department grain is requested - Returns `NULL` when category grain is requested The trade-off again though is maintenance complexity as you’ll need to manually update the routing conditions whenever measure availability changes, whereas native aggregate awareness would handle standard measures automatically (though it still wouldn’t handle level-specific measures at all, hence this more complex and manual approach. # Interested? Find out more! Rittman Analytics is a boutique data analytics consultancy that helps ambitious, digital-native businesses scale-up their approach to data, analytics and generative AI. We’re authorised delivery partners for Google Cloud along with Oracle, Segment, Cube, Dagster, Preset, dbt Labs and Fivetran and are experts at helping you design an analytics solution that’s right for your organisation’s needs, use-cases and budget and working with you and your data team to successfully implement it. If you’re looking for some help and assistance with your BI or data warehousing initiative or would just like to talk shop and share ideas and thoughts on what’s going on in the wider data world, [contact us now](https://calendly.com/markrittman/initial-discovery-call-with-mark-rittman) to organise a 100%-free, no-obligation call — we’d love to hear from you! --- # When Your Talented Data Team is Stuck in Second-Gear Canonical: https://rittmananalytics.com/blog/2025/07/30/when-your-talented-data-team-is-stuck-in-second-gear URL: https://rittmananalytics.com/blog/2025/07/30/when-your-talented-data-team-is-stuck-in-second-gear Description: You did everything right. You recognised the limits of spreadsheets, secured the budget, and invested in a talented data team. You’ve adopted the modern data stack with dbt, a cloud… Author: Mark Rittman Published: 2025-07-30T01:00:00+01:00 Modified: 2025-07-30T01:00:00+01:00 --- ![When Your Talented Data Team is Stuck in Second-Gear](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F07%2F1753885226192-9hs0g.webp%3Ffit%3D1400%252C676%26ssl%3D1) You did everything right. You recognised the limits of spreadsheets, secured the budget, and invested in a talented data team. You’ve adopted the modern data stack with dbt, a cloud warehouse such as Snowflake or BigQuery along with a self-service BI tool such as Looker and yet … something is off. The quick wins you expected are slow to materialise. Your team seems to be constantly busy, but they’re stuck in a cycle of maintenance and firefighting. Stakeholders are starting to lose faith in the data, and your team’s morale is dipping. Instead, you’re left asking why your investment in data is delivering the value you’d expected? This is a common and painful stage in a company’s data journey. It’s the moment you realise that having a data team and having a high-performing, strategic data function are two very different things. The problem isn’t your people or your technology; it’s the organisational operating system that connects them. ## **The five symptoms of a stalled data team** When a data team is struggling to transition from a support function to a strategic one, a familiar set of symptoms almost always appears: 1. **A “Flat” or “Directionless” Team :** Your daily stand-ups feel like a list of chores, not a strategic huddle. The team is reactive, fixing broken pipelines and responding to ad-hoc requests, with little time or energy left for the innovative, proactive work that truly drives the business forward. This reactive cycle is a major driver of low morale. 2. **No Single Version of the Truth :** Despite having a modern data warehouse, different teams still present conflicting numbers in meetings. The warehouse has grown organically into a confusing landscape of redundant tables and inconsistent metric definitions. This erodes the most valuable commodity in data analytics: trust. 3. **An Analytics Queue That Never Shrinks :** The data team has become a bottleneck. Simple requests for new metrics or data cuts take weeks to fulfil, frustrating business stakeholders. This forces them to either make decisions without data or create their own “shadow IT” in spreadsheets, exacerbating the data trust problem. 4. **Dashboards Without Decisions :** The team is shipping dashboards, but they don’t lead to action. There’s a disconnect between the technical work and the core business questions. The dashboards are technically correct but fail to provide the strategic insights your leadership team actually needs to make better decisions. 5. **BI Tools That Are Anything But “Self-Service”** You invested in Looker, Cube or one of the other self-service BI tools built around a semantic layer, but it’s either locked down by complex governance or has become a “wild west” of unreliable, user-generated reports. Instead of empowering business users, it’s either a bottleneck or another source of untrustworthy data. If this sounds familiar, you’re not alone. We recently partnered with **Pleo**, one of Europe’s leading FinTech unicorns, as they navigated this exact stage of their data journey and you can [read more about their story](https://www.rittmananalytics.com/case-studies/pleo) in our latest case-study on our website. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1753885226192-9hs0g.png%3Fresize%3D1400%252C676%26ssl%3D1) Their remarkable success and hyper-growth had created a new challenge: the data infrastructure and processes that had served them well during their start-up phase needed a strategic overhaul to match their future ambitions. ## **It’s Never Just a Technical Problem** Our belief is that modernising a data function is never just a technical project. Traditional consulting that focuses only on implementing the next tool often fails because it ignores the human element. Real, sustainable transformation is built on three pillars: Technology, Process, and People. Our approach is built on a proven, three-step methodology designed to address all three in tandem. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1753885225512-tzxdtr.png%3Fresize%3D1400%252C272%26ssl%3D1) **Step 1: Confront the Hard Truths (The Diagnostic)** Before you can build a better future, you must have an unflinchingly honest view of the present. We begin every modernization engagement with an intensive discovery sprint where we embed ourselves within the client’s organization. With Pleo, this meant conducting in-depth interviews with leadership and engineers, attending daily stand-ups to observe real-time dynamics, and performing a deep analysis of their existing documentation. This diagnostic phase revealed a talented, deeply committed central data team that was in need of a new operating model to support the company’s rapid growth. It allowed us to move forward with a shared, evidence-based understanding of the core challenges. **Step 2: Create the Change (The Blueprint)** With a clear diagnosis, we partner with the client’s team to co-create a new foundation. This is where we architect the technical and procedural solutions that will solve the root problems. For Pleo, this involved a multi-faceted effort: - **Co-Creating the Pleo Analytics Warehouse (PAW):** We collaborated to design a new, structured data warehouse architecture and a formal five-phase analytics development lifecycle. This brought consistency and accountability to their data product development. - **Streamlining with a Unified dbt-Looker Monorepo:** To reduce friction and improve delivery speed, we helped create a unified repository that brought dbt models and Looker assets into a single development environment. This made their workflow faster, more reliable, and easier to govern. **Step 3: Manage the Change (The Handover)** This is our key differentiator and the most critical phase. A new platform is useless if the team reverts to old habits. We actively manage the human side of the transformation to ensure new ways of working become the new standard. This means we don’t just deliver a solution and walk away. We work side-by-side with our clients’ teams, providing hands-on mentorship, overhauling team rituals, and helping to establish a culture of clear ownership. Over time, as we did with Pleo, our role shifts from hands-on building to coaching and support, as the internal team grows in confidence and takes the lead. ## **The Outcome: A Strategic, High-Performance Data Function** By focusing on the holistic system of people, processes, and technology, we were able to help Pleo accelerate the delivery of their new analytics warehouse, increase the capabilities of their internal data team, and build a more robust, scalable platform to support their continued growth. > *“Working with Rittman Analytics has been great. They have been a very helpful extension of our own Data team, rolling up their sleeves and building with us. That partnership was key to leveling-up our data foundations, which lets our team focus on what they do best: using data to drive innovation faster.”* — Pri Nagashima, VP of Data, Analytics and AI at Pleo ## **Interested? Find Out More!** Rittman Analytics is a boutique data analytics consultancy that partners with successful organisations to transform their overwhelmed data teams from a reactive cost centre into a proactive, strategic asset. We deliver more than technology; we deliver the organisational change that makes it work. You can read more about [our approach to modernising data teams and data platforms](https://www.rittmananalytics.com/services/data-team-enablement) on our website, where you can also [take our diagnostic assessment](https://markrittman.medium.com/services/data-team-enablement#diagnostic-assessment) and receive a custom playbook with targeted advice for your specific challenges. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1753885225554-xovbd.png%3Fresize%3D1400%252C348%26ssl%3D1) And of course full details on the [Pleo story](https://www.rittmananalytics.com/case-studies/pleo) and many others are on our [Case Studies](https://www.rittmananalytics.com/case-studies) page. So if you’re looking for some help and assistance scaling your data capabilities or would just like to talk shop and share your thoughts on what’s going on in your organisation and the wider data analytics world, [contact us now](https://calendly.com/markrittman/initial-discovery-call-with-mark-rittman) to organise a 100%-free, no-obligation call — we’d love to hear from you! --- # IQR-Based Website Event Anomaly Detection using Looker and Google BigQuery — Rittman Analytics Canonical: https://rittmananalytics.com/blog/2025/07/28/iqr-based-website-event-anomaly-detection-using-looker-and-google-bigquery-rittman-analytics URL: https://rittmananalytics.com/blog/2025/07/28/iqr-based-website-event-anomaly-detection-using-looker-and-google-bigquery-rittman-analytics Description: Website event tracking bugs can become costly quickly. A deployment of a new website version that triggers events multiple times can increase your hosting charges or worse, lead to overage… Author: Mark Rittman Published: 2025-07-28T01:00:00+01:00 Modified: 2025-07-28T01:00:00+01:00 --- ![IQR-Based Website Event Anomaly Detection using Looker and Google BigQuery — Rittman Analytics](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F07%2F1753874062120-ri2bhh.webp%3Ffit%3D1536%252C1024%26ssl%3D1) # Website event tracking bugs can become costly quickly. A deployment of a new website version that triggers events multiple times can increase your hosting charges or worse, lead to overage charges from your event collection service vendor that you’d not planned for. A persistent popup can negatively impact your conversion metrics and irritate users. A malfunctioning contact form can result in weeks of lost leads before anyone notices. Or your website could be working fine but, without you knowing, one of your products goes viral and quickly goes out of stock, something you could have detected if you’d been monitoring your product views/add-to-basket ratio at the time. But what number of page view events, product view events or product view/add-to-basket events is an “odd” number, an “anomaly”? Using fixed “upper” and “lower” bound limits for those counts and ratios means ignoring the trend of those numbers over time, and while page view and purchase events counts on regular days may be fairly predictable, it certainly wouldn’t be anomalous for those counts to be much, much higher on days such as Black Friday. We’ve encountered these types of issues working with clients and also managing our own website at [https://rittmananalytics.com](https://rittmananalytics.com/), and so we put-together an anomaly detection system in Looker that uses BigQuery SQL and the Interquartile Range (IQR) method to detect when website metrics start behaving unusually. ## **IQR-Based Anomaly Detection** Interquartile Range (IQR) method works well with website event data as it’s robust to outliers and doesn’t assume normal distribution, which is rarely true for web traffic. The IQR method works by: 1. Calculating the 25th percentile (Q1) and 75th percentile (Q3) of historical data 2. Computing the IQR as Q3 – Q1 3. Defining anomaly thresholds as Q1 – (multiplier × IQR) and Q3 + (multiplier × IQR) 4. Flagging any values that fall outside these bounds ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1753874062120-ri2bhh.png%3Fresize%3D1536%252C1024%26ssl%3D1) In our case we’re detecting anomalies for these specific event types and ratios: **Event Types:** - Page View - Button Clicked - Contact Us - ICP Modal Displayed - Displayed Popup **Key Ratios:** - ICP Modal Displayed / Page View - Displayed Popup / Page View These metrics and ratios let us know if we’re experiencing an unusually high amount of page views or other types of site behaviour from users, or if we’ve deployed a new marketing or lead-generation tool and its suddenly firing a lot more times than it should do, risking visitors to the site leaving and going elsewhere because of usability problems. ## **IQR-Based Anomaly-Detection using BigQuery SQL** Our implementation starts with SQL that aggregates daily event counts and calculates rolling statistical baselines. Here’s the core aggregation: ``` WITH base_events AS ( SELECT event_type, event_ts, DATE(event_ts) AS event_date FROM `ra-development.analytics.web_events_fact` WHERE event_type IN ( 'Page View', 'Button Clicked', 'Contact Us', 'Icp Modal Displayed', 'Displayed Popup' ) ), event_counts AS ( SELECT event_date, event_type, COUNT(*) AS event_count FROM base_events GROUP BY event_date, event_type ), pivoted_counts AS ( SELECT event_date, COALESCE(MAX(CASE WHEN event_type = 'Page View' THEN event_count END), 0) AS page_view_count, COALESCE(MAX(CASE WHEN event_type = 'Button Clicked' THEN event_count END), 0) AS button_clicked_count, COALESCE(MAX(CASE WHEN event_type = 'Contact Us' THEN event_count END), 0) AS contact_us_count, COALESCE(MAX(CASE WHEN event_type = 'Icp Modal Displayed' THEN event_count END), 0) AS icp_modal_displayed_count, COALESCE(MAX(CASE WHEN event_type = 'Displayed Popup' THEN event_count END), 0) AS displayed_popup_count FROM event_counts GROUP BY event_date ) ``` The key insight here is that we’re pivoting our event data from a long format (one row per event type per day) to a wide format (one row per day with columns for each event type). This makes the subsequent ratio calculations much cleaner. Next, we calculate our key ratios using BigQuery’s `SAFE_DIVIDE` function to handle division by zero gracefully: ``` event_ratios AS ( SELECT *, CASE WHEN page_view_count != 0 THEN SAFE_DIVIDE(icp_modal_displayed_count, page_view_count) ELSE NULL END AS icp_modal_displayed_ratio, CASE WHEN page_view_count != 0 THEN SAFE_DIVIDE(displayed_popup_count, page_view_count) ELSE NULL END AS displayed_popup_ratio FROM pivoted_counts ) ``` BigQuery’s `APPROX_QUANTILES` function returns an array of quantiles, making it easy to extract Q1 and Q3: ``` ratio_stats AS ( SELECT r1.event_date, APPROX_QUANTILES(r2.icp_modal_displayed_ratio, 4)[OFFSET(1)] AS icp_modal_displayed_ratio_q1, APPROX_QUANTILES(r2.icp_modal_displayed_ratio, 4)[OFFSET(3)] AS icp_modal_displayed_ratio_q3, APPROX_QUANTILES(r2.displayed_popup_ratio, 4)[OFFSET(1)] AS displayed_popup_ratio_q1, APPROX_QUANTILES(r2.displayed_popup_ratio, 4)[OFFSET(3)] AS displayed_popup_ratio_q3 FROM event_ratios r1 JOIN event_ratios r2 ON r2.event_date BETWEEN DATE_SUB(r1.event_date, INTERVAL 371 DAY) AND DATE_SUB(r1.event_date, INTERVAL 1 DAY) WHERE r2.icp_modal_displayed_ratio IS NOT NULL OR r2.displayed_popup_ratio IS NOT NULL GROUP BY r1.event_date ) ``` The self-join in the above CTE lets us look-back 371 days (a year plus the maximum variation in days between Black Friday from one year to another) to establish a baseline, excluding the current day. For us we’re detecting anomalies in daily metrics but you could increase the granularity to hourly if a faster response to events is needed. ### **Dynamic Anomaly Thresholds with LookML Parameters** All of this SQL is going to run as a single LookML derived table definition in a LookML view, like this: ``` view: anomaly_detection { view_label: "Daily Anomaly Detection - Web Events" derived_table: { sql: WITH base_events AS ( SELECT event_type, event_ts, DATE(event_ts) AS event_date, ... ``` and as we’re running this SQL within a LookML view, we also now have the ability to introduce parameters into the view and the derived table SQL to enable us to dynamically adjust the thresholds for the upper and lower bounds in our ICQ calculations. ``` -- ICP Modal Displayed Ratio Anomaly Detection (rs.icp_modal_displayed_ratio_q1 - {% parameter lower_bound_multiplier %} * (rs.icp_modal_displayed_ratio_q3 - rs.icp_modal_displayed_ratio_q1)) AS icp_modal_displayed_ratio_lower_bound, (rs.icp_modal_displayed_ratio_q3 + {% parameter upper_bound_multiplier %} * (rs.icp_modal_displayed_ratio_q3 - rs.icp_modal_displayed_ratio_q1)) AS icp_modal_displayed_ratio_upper_bound, CASE WHEN r.icp_modal_displayed_ratio < (rs.icp_modal_displayed_ratio_q1 - {% parameter lower_bound_multiplier %} * (rs.icp_modal_displayed_ratio_q3 - rs.icp_modal_displayed_ratio_q1)) OR r.icp_modal_displayed_ratio > (rs.icp_modal_displayed_ratio_q3 + {% parameter upper_bound_multiplier %} * (rs.icp_modal_displayed_ratio_q3 - rs.icp_modal_displayed_ratio_q1)) THEN TRUE ELSE FALSE END AS is_icp_modal_displayed_ratio_anomalous ``` This approach allows users to adjust the sensitivity of anomaly detection in real-time without modifying the underlying SQL, via three parameters we define in the LookML view following the derived table SQL query definition, as shown below: ``` parameter: anomaly_metric { allowed_value: {label: "Page View Count" value: "PAGE_VIEW_COUNT"} allowed_value: {label: "Button Clicked Count" value: "BUTTON_CLICKED_COUNT"} allowed_value: {label: "Contact Us Count" value: "CONTACT_US_COUNT"} allowed_value: {label: "ICP Modal Displayed Count" value: "ICP_MODAL_DISPLAYED_COUNT"} allowed_value: {label: "Displayed Popup Count" value: "DISPLAYED_POPUP_COUNT"} allowed_value: {label: "ICP Modal Display Ratio" value: "ICP_MODAL_DISPLAYED_RATIO"} allowed_value: {label: "Displayed Popup Ratio" value: "DISPLAYED_POPUP_RATIO"} default_value: "PAGE_VIEW_COUNT" type: unquoted label: "Metric Selector" group_label: "Parameters" description: "Choose which metric (count or ratio) to evaluate for anomalies." } parameter: upper_bound_multiplier { allowed_value: {label: "1.0" value: "1.0"} allowed_value: {label: "1.5" value: "1.5"} allowed_value: {label: "2.0" value: "2.0"} allowed_value: {label: "2.5" value: "2.5"} default_value: "1.5" type: unquoted label: "Upper Bound Multiplier" group_label: "Parameters" description: "Multiplier applied to the IQR to calculate the upper threshold for anomaly detection." } parameter: lower_bound_multiplier { allowed_value: {label: "0.0" value: "0.0"} allowed_value: {label: "0.5" value: "0.5"} allowed_value: {label: "1.0" value: "1.0"} allowed_value: {label: "1.5" value: "1.5"} allowed_value: {label: "2.0" value: "2.0"} allowed_value: {label: "2.5" value: "2.5"} default_value: "1.5" type: unquoted label: "Lower Bound Multiplier" group_label: "Parameters" description: "Multiplier applied to the IQR to calculate the lower threshold for anomaly detection." } ``` These parameters work together with dynamic measures that adjust based on the selected metric, like so: ``` measure: metric { label_from_parameter: anomaly_metric description: "Total value of the selected metric used for anomaly detection." group_label: "Anomaly Metrics" type: sum sql: ${TABLE}.{% parameter anomaly_metric %} ;; } dimension: is_metric_anomalous { label: "Is Anomalous?" description: "Flag indicating if the selected metric falls outside the calculated bounds." group_label: "Anomaly Metrics" type: yesno sql: ${TABLE}.IS_{% parameter anomaly_metric %}_ANOMALOUS ;; } ``` The full implementation for the LookML view is available as a Github Gist [here](https://gist.github.com/markrittman/b44526efd7511020691cabdc143db801). ## **Analyzing Historical Anomalies as Trends** Now that we’ve setup the LookML view and separately, added it as an explore in your LookML model, you can open-up the Explore interface in Looker and start to analyze the history of anomalies in your event data. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1753874062123-b4f3fg.png%3Fresize%3D3095%252C2277%26ssl%3D1) As you can see in the screenshot above, the parameters we defined in the LookML view become filters that let us select: - **Metric Selector**: Set to “Page View Count” to analyze page view anomalies - **Lower Bound Multiplier**: Set to 1.5 (this determines how sensitive we are to low outliers) - **Upper Bound Multiplier**: Set to 1.5 (this determines how sensitive we are to high outliers) The great thing about this parameterized approach is that you can adjust these values in real-time to fine-tune your anomaly detection sensitivity. A multiplier of 1.5 is moderately sensitive – values of 1.0 will catch more anomalies (potentially including false positives), while values of 2.0+ will only flag the most extreme outliers. For our analysis, we’re pulling: - **Event Date Date** (as our dimension) - **Metric** (which will display as “Page View Count” based on our parameter selection) - **Metric Lower Bound** and **Metric Upper Bound** (for visualization) - **Is Anomalous?** (our boolean flag for anomaly detection) Displaying this anomaly data visually though really makes it clear when your chosen metric is reporting a value way-outside of what’s expected for that day. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1753874063965-6uscrp.png%3Fresize%3D3096%252C2277%26ssl%3D1) The line chart visualization in the screenshot above shows: - **Blue line**: The actual Page View Count over time - **Gray lines**: The upper and lower bounds for anomaly detection - **Data points**: Each point represents a daily aggregate In this particular example, you can see there was a significant spike in page views around April 14-28, where the actual values exceeded the upper bound threshold. - **Baseline period**: From March through early April, page views were consistently low and stable - **Anomalous spike**: The April spike stands out dramatically against the historical baseline - **Return to normal**: By May, traffic patterns had returned to baseline levels And if you’re interested in what caused these spikes in page views, all from the same IP address and with browser IDs that changed every 10 or 20 page views thereby leading to our Segment MTU numbers going past our monthly allowance? It was bot traffic scraping every page from our site, every hour, something we only managed to halt by by using Cloudflare’s Web Application Firewall (WAF) feature. ## **Building a Anomaly-Monitoring Dashboard** While the Explore feature is excellent for ad-hoc analysis, you can also build a monitoring dashboard that gives you an at-a-glance view of all your critical anomaly-detection metrics. In the screenshot below you can see a simple example of an Anomaly Detection Dashboard in Looker, to which you could also add alerts to tell us, for example, when page views or popup display frequency ratios yesterday were anomalous compared to the norm. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1753874064351-d2bghe.png%3Fresize%3D3096%252C2159%26ssl%3D1) ## **Interested? Find Out More!** Rittman Analytics is a boutique data analytics consultancy that helps ambitious, digital-native businesses scale-up their approach to data, analytics and generative AI. We’re authorised delivery partners for [Google Cloud](https://www.rittmananalytics.com/partners/looker) along with [Oracle](https://www.rittmananalytics.com/partners/oracle), [Segment](https://www.rittmananalytics.com/partners/segment), [Cube](https://www.rittmananalytics.com/partners/cube), [Dagster](https://www.rittmananalytics.com/partners/dagster), [Preset](https://www.rittmananalytics.com/partners/preset), [dbt Labs](https://www.rittmananalytics.com/partners/dbt-labs) and [Fivetran](https://www.rittmananalytics.com/partners/fivetran) and are experts at helping you design an analytics solution that’s right for your organisation’s needs, use-cases and budget and working with you and your data team to successfully implement it. If you’re looking for some help and assistance with your Looker implementation or website event tracking or would just like to talk shop and share ideas and thoughts on what’s going on in your organisation and the wider data analytics world, [contact us now](https://calendly.com/markrittman/initial-discovery-call-with-mark-rittman?utm_source=medium.com&utm_medium=blog&utm_campaign=customer_segmentation) to organise a 100%-free, no-obligation call — we’d love to hear from you! [](https://rittmananalytics.com/blog?author=5846da08f5e231d83261e221) --- # Integrating Looker and LangChain with langchain-looker-agent Canonical: https://rittmananalytics.com/blog/2025/06/02/integrating-looker-and-langchain-with-langchain-looker-agent URL: https://rittmananalytics.com/blog/2025/06/02/integrating-looker-and-langchain-with-langchain-looker-agent Description: We’ve just released a prototype of a new Langchain tool that enables LangChain applications and agentic workflows to connect to Looker, via its Open SQL Interface. In the past when… Author: Mark Rittman Published: 2025-06-02T16:47:00+00:00 Modified: 2025-06-02T16:47:00+00:00 --- ![Integrating Looker and LangChain with langchain-looker-agent](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2F110e5-a63820d4-6b37-4fb3-924f-52508023e9b4_1000x897.png%3Fssl%3D1) We’ve just released a [prototype of a new Langchain tool](https://pypi.org/project/langchain-looker-agent/) that enables LangChain applications and agentic workflows to connect to Looker, via its [Open SQL Interface](https://cloud.google.com/looker/docs/sql-interface). In the past when we’ve worked on [AI-powered chatbots](https://www.rittmananalytics.com/blog/2024/7/5/ai-powered-conversational-data-analyst-chatbot), [dashboard narrative generators](https://www.rittmananalytics.com/blog/2024/9/30/creating-a-self-learning-data-analytics-sql-chatbot-using-langchain-rag-open-ai-and-google-bigquery-vector-store) and [data apps that use LangChain](https://www.rittmananalytics.com/blog/2024/9/17/genai-profit-and-loss-report-analysis) to orchestrate and connect LLMs such as Open AI GPT-4o and Google Gemini to data in our data warehouse, we’d ideally like to have routed that connection through our Looker semantic layer instead rather than directly to the warehouse table schema. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2F110e5-a63820d4-6b37-4fb3-924f-52508023e9b4_1000x897.png%3Fssl%3D1) When AI systems query databases directly, they’re working with raw schemas without business context. This means different applications might interpret the same data differently, leading to inconsistent results across your organization. More importantly, AI agents have to infer business logic from technical schemas, which can lead to technically correct but business-meaningless results. Looker’s LookML semantic layer, in-contrast, provides pre-validated business logic and standardized metric definitions. Instead of asking an AI to figure out what “revenue” means from database field names, the semantic layer provides explicit business definitions and calculations, reducing the risk of hallucinations and answering the user’s query using the wrong set of metrics and dimension values. #### **Integrating Looker with LangChain** The prototype langchain-looker-agent we’ve developed was designed to be as familiar to work with as possible for anyone who’s used other [Langchain SQL Toolkits](https://python.langchain.com/docs/integrations/tools/sql_database/), as shown in the code example below. ``` from langchain_looker_agent import create_looker_sql_agent, LookerSQLDatabase from langchain_openai import ChatOpenAI # Initialize Looker connection looker_db = LookerSQLDatabase( looker_instance_url=os.getenv('LOOKER_BASE_URL'), lookml_model_name=os.getenv('LOOKML_MODEL_NAME'), client_id=os.getenv('LOOKER_CLIENT_ID'), client_secret=os.getenv('LOOKER_CLIENT_SECRET'), jdbc_driver_path=os.getenv('JDBC_DRIVER_PATH') ) # Create the agent llm = ChatOpenAI(model="gpt-4o", temperature=0) toolkit = LookerSQLToolkit(db=looker_db, llm=llm) agent = create_looker_sql_agent(llm=llm, toolkit=toolkit) ``` The [langchain-looker-agent](https://pypi.org/project/langchain-looker-agent/) tool maintains conversation memory, supports follow-up questions, and includes explore discovery capabilities. Importantly, all responses are grounded in your validated business logic rather than AI interpretations of raw data. [Key components](https://github.com/rittmananalytics/langchain-looker-sql-agent/blob/main/src/langchain_looker_agent/agent.py) within the langchain-looker-agent package are: - **LookerSQLDatabase**: A custom class that wraps the Looker JDBC connection, mimicking the interface of LangChain’s SQLDatabase utility. It handles connecting to Looker, fetching metadata (Explores as tables, Fields as columns) and executing SQL queries. It’s specifically designed to work with Looker’s SQL dialect (Calcite) including backticked identifiers and AGGREGATE() for measures. - **LookerSQLToolkit**: A LangChain toolkit that bundles database interaction tools (list tables/Explores, get schema, query database) using the LookerSQLDatabase. - **create\_looker\_sql\_agent**: A factory function to create a LangChain ReAct agent pre-configured with the LookerSQLToolkit and a specialized system prompt guiding the LLM on Looker’s specific SQL syntax and data structures. The [Github repo](https://github.com/rittmananalytics/langchain-looker-sql-agent) that accompanies the PyPI package download comes with quickstart example Python code and two Jupyter notebooks that provide further code examples that will help you get started, and we’ve also put-together an [example Looker Chatbot web application](https://github.com/rittmananalytics/langchain-looker-agent-demo-chatbot) in another Github repo to show the tool being used in a more complex use-case. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2F36645-cc885169-7fc0-43a4-9175-624dda13c528_1000x881.png%3Fssl%3D1) #### **How Does the Integration Work?** As the [Looker Open SQL Interface](https://cloud.google.com/looker/docs/sql-interface) uses JDBC to connect to Looker it requires a Java JDK to be downloaded and installed along with the [Avatica/Calcite JDBC driver JAR file](https://github.com/looker-open-source/calcite-avatica/releases) that’s [hosted for download](https://cloud.google.com/looker/docs/sql-interface#download_jdbc_driver) on the Looker open-source Github repository. ``` sudo apt-get update sudo apt-get install -y openjdk-11-jdk --no-install-recommends java -version # Verify installation export JAVA_HOME=$(java -XshowSettings:properties -version 2>&1 > /dev/null | grep 'java.home' | awk '{print $3}') echo $JAVA_HOME mkdir drivers cd drivers wget https://github.com/looker-open-source/calcite-avatica/releases/download/avatica-1.26.0-looker/avatica-1.26.0-looker.jar ``` An important consideration to note with this dependency on a Java JDK is that it precludes the use of serverless environments such as Google Cloud Run Functions to run this package, due to the restrictions they place on installing third-party packages such as a Java JDK. Our recommended approach instead is to use [Cloud Run](https://cloud.google.com/run?hl=en) that provides containerized environments where you can install multiple language runtimes within a single container, and allows you to create a Docker image that includes both Python for your application code and Java for the JDBC driver requirements, as we do when deploying AI apps using the langchain-looker-agent package: ``` FROM python:3.10-slim RUN apt-get update && \ apt-get install -y wget openjdk-17-jre-headless && \ rm -rf /var/lib/apt/lists/* ENV JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64 WORKDIR /app COPY cloud-function/requirements.txt ./ RUN pip install --no-cache-dir -r requirements.txt RUN mkdir drivers && \ cd drivers && \ wget https://github.com/looker-open-source/calcite-avatica/releases/download/avatica-1.26.0-looker/avatica-1.26.0-looker.jar ENV LOOKER_JDBC_DRIVER_PATH=/app/drivers/avatica-1.26.0-looker.jar COPY cloud-function/ . ENV PORT=8080 CMD ["functions-framework", "--target=hello_http", "--host", "0.0.0.0", "--port", "8080"] ``` #### **Why Connect Looker to LangChain and its EcoSystem?** Like the forthcoming official [Conversational Analytics API](https://cloud.google.com/gemini/docs/conversational-analytics-api/overview) coming soon from Google, this new AI integration for Looker enables businesses build AI applications that maintain conversation memory across interactions and handle cyclical reasoning patterns that go far beyond simple question-and-answer scenarios. But by integrating with LangChain’s extensive library of [over 700 integrations](https://python.langchain.com/docs/integrations/providers/) with external tools, databases, and APIs along with enterprise-grade features like [LangSmith](https://docs.smith.langchain.com/) for observability and evaluation and [LangGraph](https://langchain-ai.github.io/langgraph/tutorials/introduction/) for more complex agentic workflows, it also now opens-up use-cases such as: - **Multi-Agent Analytical Workflows**: Build sophisticated data analysis agents, for example using the open-source agentic orchestration tool n8n, that can coordinate multiple specialized sub-agents (data validators, trend analyzers, report generators) working together on complex analytical tasks, rather than limiting interactions to single-query responses - **Autonomous Decision-Making Pipelines**: You can create agents that can automatically trigger business actions based on Looker insights, such as inventory reordering systems that analyze sales trends and automatically generate purchase orders through connected ERP systems - **Cross-Platform Data Orchestration**: You could also combine Looker’s semantic layer with data from CRM systems, external APIs, and real-time feeds to create comprehensive business intelligence agents that provide holistic insights rather than siloed analytics - **Enterprise-Grade Scalability**: Importantly, you’ll be able to leverage LangChain’s production-ready infrastructure for deploying conversational analytics across thousands of users while maintaining governance, monitoring, and evaluation capabilities through LangSmith ### **INTERESTED? FIND-OUT MORE** The [langchain-looker-agent](https://pypi.org/project/langchain-looker-agent/), available now as a preview/prototype release suitable for testing and non-production use-cases as of today, is available as a pip-installable package on PyPI and includes comprehensive documentation on GitHub. You’ll need Looker API credentials, a JDBC driver, and your LookML models properly configured. - [https://pypi.org/project/langchain-looker-agent/](https://pypi.org/project/langchain-looker-agent/) The source code for the langchain-looker-agent tool is available for access and cloning on our Github site, along with the source code for a demo chatbot app built using Python, Replit and the langchain-looker-agent tool. - [https://github.com/rittmananalytics/langchain-looker-sql-agent](https://github.com/rittmananalytics/langchain-looker-sql-agent) - [https://github.com/rittmananalytics/langchain-looker-agent-demo-chatbot](https://github.com/rittmananalytics/langchain-looker-agent-demo-chatbot) If you’re interested in implementing AI-powered analytics with semantic layer integration using Looker or Cube, we’d be happy to help. Rittman Analytics is a boutique data analytics consultancy that helps ambitious, digital-native businesses scale-up their approach to data, analytics and [generative AI](https://www.rittmananalytics.com/services/genai). We’re authorised delivery partners for [Google Cloud](https://www.rittmananalytics.com/partners/looker) along with [Oracle](https://www.rittmananalytics.com/partners/oracle), [Segment](https://www.rittmananalytics.com/partners/segment), [Cube](https://www.rittmananalytics.com/partners/cube), [Dagster](https://www.rittmananalytics.com/partners/dagster), [Preset](https://www.rittmananalytics.com/partners/preset), [dbt Labs](https://www.rittmananalytics.com/partners/dbt-labs) and [Fivetran](https://www.rittmananalytics.com/partners/fivetran) and are experts at helping you design an analytics solution that’s right for your organisation’s needs, use-cases and budget and working with you and your data team to successfully implement it. If you’re looking for some help and assistance with your AI initiative or would just like to talk shop and share ideas and thoughts on what’s going on in your organisation and the wider data analytics world, [contact us now](https://calendly.com/markrittman/initial-discovery-call-with-mark-rittman?utm_source=medium.com&utm_medium=blog&utm_campaign=customer_segmentation) to organise a 100%-free, no-obligation call — we’d love to hear from you! --- # Introducing Conversational Analytics, Data Agents and Code Interpreter in Looker and Looker Studio Canonical: https://rittmananalytics.com/blog/2025/05/11/introducing-conversational-analytics-data-agents-and-code-interpreter-in-looker-and-lookernbspstudio URL: https://rittmananalytics.com/blog/2025/05/11/introducing-conversational-analytics-data-agents-and-code-interpreter-in-looker-and-lookernbspstudio Description: Conversational Analytics is Google’s implementation of AI-powered natural language querying for Looker and Looker Studio. At its core, it enables users to interact with their data using everyday language rather… Author: Mark Rittman Published: 2025-05-11T22:42:22+00:00 Modified: 2025-05-11T22:42:22+00:00 --- ![Introducing Conversational Analytics, Data Agents and Code Interpreter in Looker and Looker Studio](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2F816ba-8f6cc46c-e352-4000-97a5-ca3022cf32c1_1000x503.png%3Fssl%3D1) Conversational Analytics is Google’s implementation of AI-powered natural language querying for Looker and Looker Studio. At its core, it enables users to interact with their data using everyday language rather than having to learn SQL or navigate complex dashboard interfaces. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2F816ba-8f6cc46c-e352-4000-97a5-ca3022cf32c1_1000x503.png%3Fssl%3D1) The goal for conversational analytics is to further democratise data access. By allowing users to simply type questions like “What were our top-selling products last quarter?” or “Show me website traffic trends by channel over the past six months,” Conversational Analytics is like having a data analyst available 24/7 who already understands your business context. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2F41c5d-6f5bec97-50ab-48fc-ab34-73fa50a2e5d4_1000x733.png%3Fssl%3D1) What’s particularly powerful is how Conversational Analytics leverages Looker’s semantic layer. This ensures that everyone in the organization is using the same metric definitions and business rules, eliminating the “multiple versions of the truth” problem that plagues many analytics implementations ### Chat with Looker’s Trusted Metrics Store In Looker, once your Looker admin has [enabled Gemini in Looker and granted you the Gemini role](https://cloud.google.com/looker/docs/studio/conversational-analytics-looker), you can access Conversational Analytics from [either the Looker homepage or from an explore (analysis) page](https://cloud.google.com/looker/docs/studio/conversational-analytics-looker), and in all cases you pick a particular Looker explore as the starting point for your conversation. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2F74992-93d3883b-56a5-4b67-8a52-a25011e7d432_1000x789.png%3Fssl%3D1) From that point on and as shown in the screenshot before, you can carry on a typical questions-and-answers conversation with the data exposed by your explore, with the history of previous questions and answers within that particular conversation available as context for whatever question you answer next. ### Creating Conversational Analytics Data Agents Although the Looker semantic layer provides a more governed and descriptive chatbot datasource than you’d typically get with a database schema, there still could be a number of different metrics with “revenue” in their name when your business always reports using net revenue, for example. Or there could be specific calculation formulae that the Conversational Analytics wouldn’t know about by default, and of course Conversational Analytics doesn’t know the context for your questions or the type of analyst role you’re expecting it to adopt when answering your questions. Conversational Analytics also therefore gives you the ability to create customized [data agents](https://cloud.google.com/looker/docs/studio/conversational-data-agents?hl=en) for specific use cases. These agents are essentially specialized assistants that combine domain-specific knowledge with data access. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2Fe170d-173e0311-3123-48b3-88d1-f5b1288135b3_1000x346.png%3Fssl%3D1) To create a data agent: 1. Navigate to Conversational Analytics in Looker Studio or Looker 2. Press **\+ Create agent** in the **\+ Create Conversation** section 3. Provide an agent name and description that clearly explains its purpose 4. Connect to a data source (BigQuery table, Looker Explore, etc.) 5. Add instructions to help the agent understand your data context The instructions field is where you customize the agent’s understanding of your data, and is crucial for getting accurate and relevant responses. Some effective instruction types include: - *Synonyms: “If someone says ‘revenue’, use the ‘total\_sales’ field”* - *Key fields: “The most important fields are order\_id, customer\_id, and order\_date”* - *Filtering guidance: “Unless stated otherwise, always filter data for the current year”* - *Business definitions: “We consider ‘loyal’ customers to be those with more than 5 orders”* An approach I’ve been using to create these instructions is to make use of the LookML you can export from a KPI dashboard and have Vertex AI create the instructions using that LookML and a screenshot of the KPIs in the dashboard, as a reference: ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2F68d2c-4de3f604-5d37-4774-aee6-6e6d12fd3089_1000x737.png%3Fssl%3D1) “Priming” an agent in this way gives it a lot more context and guidance on what fields in the explore are important to you, making its answers to your questions more relevant and correct. Using these instructions and the data from your Looker explore or Looker Studio data source, Conversational Analytics is particularly good at handling questions that can be answered with a single visualization: - Metric trends over time (“How have sales changed over the past 12 months?”) - Distributions (“What’s the breakdown of orders by product category?”) - Top/bottom performers (“Show me our top 10 customers by revenue”) - Simple comparisons (“Compare website traffic across our main channels”) - Basic aggregations (“What’s our average order value?”) As with any new technology, there are some limitations to be aware of: - **Data limits**: For Looker data, queries are limited to returning a maximum of 5,000 rows - **Visualization constraints**: While basic chart types are well-supported, more complex visualizations like heat maps may have inconsistent behavior - **Question complexity**: More advanced analytical questions (period-over-period analysis, statistical analysis) aren’t supported in the base implementation - **BigQuery limitations**: Conversational Analytics doesn’t support BigQuery’s Flexible Column Names feature ### Advanced Analytics using Code Interpreter If you’re looking to push beyond the standard capabilities, Looker Studio (like tools such as Claude and ChatGPT) offers an experimental [Code Interpreter](https://cloud.google.com/looker/docs/studio/conversational-analytics-code-interpreter?hl=en) feature that significantly extends what’s possible with Conversational Analytics. Code Interpreter translates natural language questions into Python code and executes that code to provide advanced analysis and visualizations. This bridges the gap between the simple SQL-based queries that Looker and Looker Studio normally generate and more sophisticated data science techniques. To enable the Code Interpreter: 1. Navigate to **Conversational Analytics** in Looker Studio 2. In the left navigation panel, click the **Advanced analytics** toggle As you can see in the screenshot below you can also choose to view the setup instructions the Code Interpreter feature is sending to its Python execution environment, and the resulting depth of analysis provided by the Conversational Analytics agent is now greater than with Conversational Analytics answers. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2Fbe9ab-3f51aa7d-e0b3-4583-9323-60e3c3801820_1000x568.png%3Fssl%3D1) With Code Interpreter enabled, the range of questions you can answer expands dramatically to include: - Period-over-period comparisons (“How do sales this year compare to last year?”) - Identifying outliers and anomalies (“Find unusual patterns in our website traffic”) - Cohort analysis for understanding customer retention - Calculating compound annual growth rates by product category - More sophisticated visualizations using libraries like matplotlib and seaborn The Code Interpreter supports numerous Python libraries including pandas, numpy, scikit-learn, and tensorflow, enabling advanced statistical analysis and even machine learning techniques. ### INTERESTED? FIND OUT MORE! Rittman Analytics is a boutique data analytics consultancy that helps ambitious, digital-native businesses scale-up their approach to data, analytics and [generative AI](https://www.rittmananalytics.com/services/genai). We’re authorised delivery partners for [Google Cloud](https://www.rittmananalytics.com/partners/looker) along with [Oracle](https://www.rittmananalytics.com/partners/oracle), [Segment](https://www.rittmananalytics.com/partners/segment), [Cube](https://www.rittmananalytics.com/partners/cube), [Dagster](https://www.rittmananalytics.com/partners/dagster), [Preset](https://www.rittmananalytics.com/partners/preset), [dbt Labs](https://www.rittmananalytics.com/partners/dbt-labs) and [Fivetran](https://www.rittmananalytics.com/partners/fivetran) and are experts at helping you design an analytics solution that’s right for your organisation’s needs, use-cases and budget and working with you and your data team to successfully implement it. If you’re looking for some help and assistance with your AI initiative or would just like to talk shop and share ideas and thoughts on what’s going on in your organisation and the wider data analytics world, [contact us now](https://calendly.com/markrittman/initial-discovery-call-with-mark-rittman?utm_source=medium.com&utm_medium=blog&utm_campaign=customer_segmentation) to organise a 100%-free, no-obligation call — we’d love to hear from you! --- # London Looker Meetup – May 8th 2025 – Registration Open & Call for Speakers! Canonical: https://rittmananalytics.com/blog/2025/03/31/london-looker-meetup-may-8th-2025 URL: https://rittmananalytics.com/blog/2025/03/31/london-looker-meetup-may-8th-2025 Description: The first London Looker Meetup will be on Thursday, May 8th at the Google London — Pancras Square office from 17:30–20:30 and registration is now open! Back in 2019 Jonathan… Author: Mark Rittman Published: 2025-03-31T12:49:42+00:00 Modified: 2025-03-31T12:49:42+00:00 --- ![London Looker Meetup – May 8th 2025 – Registration Open & Call for Speakers!](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2Ff85d3-9e0bf430-312b-4669-84c0-3fa0a88936a4_1340x752.png%3Fssl%3D1) The first London Looker Meetup will be on Thursday, May 8th at the [Google London — Pancras Square office](https://g.co/kgs/EuHrz27) from 17:30–20:30 and [registration is now open!](https://www.meetup.com/london-looker-meetup/events/306979719/?eventOrigin=group_upcoming_events) ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2Ff85d3-9e0bf430-312b-4669-84c0-3fa0a88936a4_1340x752.png%3Fssl%3D1) Back in 2019 Jonathan Palmer and I ran the [Looker Dev Meetup](https://trumpet-maroon-65wz.squarespace.com/blog/2019/07/01/news-on-the-second-london-looker-developer-meetup-10th-july-2019-at-gocardless-london) in London, with the aim then and now being create a space for showcasing and discussing what data people in London are building in and around the Looker ecosystem. The pandemic led to that meetup group stopping, but I’m now looking to restart a similar meetup group that’s community-run and organized and with the support of Google and the partner community, with the [first meeting happening this coming May 8th](https://www.meetup.com/london-looker-meetup/events/306979719/?eventOrigin=group_upcoming_events) with Google kindly hosting the first event at their Pancras Square offices. We’re organising the agenda right now and so if you or someone you know has a great story to tell about what you’ve been doing with Looker, or maybe share some tips or techniques on how to get the best from Looker and Looker Studio, [contact me via email](mailto:mark.rittman@rittmananalytics.com) or [fill-in this form](https://forms.gle/TU6sHafTikMtiD386) and we’ll get back to you shortly. Thanks for reading Rittman Analytics Blog! Subscribe for free to receive new posts and support my work. --- # Drill to Detail Podcast Transcripts & Gen AI Insights Now Available, Powered by OpenAI Whisper API & BigQuery Colab Notebooks Canonical: https://rittmananalytics.com/blog/2025/02/14/drill-to-detail-whisper-api-colab-notebooks URL: https://rittmananalytics.com/blog/2025/02/14/drill-to-detail-whisper-api-colab-notebooks Description: We’ve been asked many times in the past for episode transcripts for the Drill to Detail Podcast, and I’m therefore pleased to launch our new Drill to Detail Podcast website… Author: Mark Rittman Published: 2025-02-14T22:24:21+00:00 Modified: 2025-02-14T22:24:21+00:00 --- ![Drill to Detail Podcast Transcripts & Gen AI Insights Now Available, Powered by OpenAI Whisper API & BigQuery Colab Notebooks](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2Fe604e-07ae4298-00be-4b00-9377-eab236288bd0_1000x1053.png%3Fssl%3D1) We’ve been asked many times in the past for episode transcripts for the Drill to Detail Podcast, and I’m therefore pleased to launch our new [Drill to Detail Podcast website](https://podcasts.rittmananalytics.com/) where you can not-only listen to all our past episodes but also view transcripts along with analysis and key quotes from the episode’s special guest. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2Fe604e-07ae4298-00be-4b00-9377-eab236288bd0_1000x1053.png%3Fssl%3D1) Until recently doing episode transcription manually or through one of the first-generation online services was a long and tedious process but with services such as [OpenAI’s Whisper API](https://openai.com/index/whisper/) service now available that use generative AI technology, we created a [BigQuery Colab Enterprise notebook](https://cloud.google.com/bigquery/docs/notebooks-introduction) that not only transcribed all of our past episodes but also generated episode summaries, lists of key points and speaker quotes that we wrote back to a BigQuery table and used as the data source for this new web app. #### Leveraging BigQuery Colab Notebooks and the OpenAI Whisper API If you’re interested in how we pulled these transcripts and episode insights together, the full python code can be found in [this BigQuery notebook](https://github.com/rittmananalytics/podcast_transcriber_openai_whisper_bigquery/transcribe_podcast_feed_bigquery_notebook.ipynb) and, skipping-over the initial imports and setup of the BigQuery and OpenAI API clients we started first by defining functions that take a given podcast episode URL and download the episode MP3 file, then split that file into chunks to fit within the Whisper API file size limit. ``` def download_mp3(url, filename): response = requests.get(url) with open(filename, 'wb') as f: f.write(response.content) def split_audio(filename, chunk_length_ms=60000): # 1 minute chunks audio = AudioSegment.from_mp3(filename) chunks = [] for i in range(0, len(audio), chunk_length_ms): chunk = audio[i:i+chunk_length_ms] chunk_file = f"{filename}_chunk_{i//chunk_length_ms}.mp3" chunk.export(chunk_file, format="mp3") chunks.append(chunk_file) return chunks ``` The next function performs the actual transcription, passing the mp3 file chunks to the Whisper API and returning the speech transcription. ``` def transcribe_audio(client, filename): with open(filename, "rb") as audio_file: transcript = client.audio.transcriptions.create( model="whisper-1", file=audio_file ) return transcript.text ``` And now the main body of code for the notebook sets-up modules to parse the RSS feed that we pass to the notebook in order to extract episode titles, links, descriptions and other metadata, process each episode in the feed and assemble the full episode transcription from all each episode’s chunk transcriptions. ``` def process_podcast(url): feed = feedparser.parse(url) episodes = [] for entry in feed.entries: episode = { 'title': entry.get('title', ''), 'link': entry.get('link', ''), 'description': entry.get('description', ''), 'published': entry.get('published', ''), } audio_url = None if 'links' in entry: for link in entry.links: if 'type' in link and link['type'].startswith('audio/'): audio_url = link.get('href') break if not audio_url and 'enclosures' in entry: for enclosure in entry.enclosures: if 'type' in enclosure and enclosure['type'].startswith('audio/'): audio_url = enclosure.get('href') break if audio_url: episode['audio_url'] = audio_url episodes.append(episode) return episodes def process_episode(client, episode): print(f"Processing episode: {episode['title']}") filename = f"temp_{episode['title'].replace(' ', '_')}.mp3" download_mp3(episode['audio_url'], filename) chunks = split_audio(filename) full_transcript = "" for chunk in chunks: transcript = transcribe_audio(client, chunk) full_transcript += transcript + " " os.remove(chunk) classification = classify_episode(client, episode['title']) guest_name = episode['title'].split('with')[-1].strip() if 'with' in episode['title'] else "Guest" labeled_transcript = label_speakers(client, full_transcript, guest_name) summary_and_insights = generate_summary_and_insights(client, labeled_transcript, guest_name) episode['transcript'] = labeled_transcript episode['classification'] = classification episode['summary_and_insights'] = summary_and_insights os.remove(filename) # Write the processed episode to BigQuery write_to_bigquery(episode) return episode def main(feed_url, num_episodes): client = openai.OpenAI(api_key=' max_chunk_size: chunks.append(' '.join(current_chunk)) current_chunk = [word] current_size = len(word) else: current_chunk.append(word) current_size += len(word) + 1 if current_chunk: chunks.append(' '.join(current_chunk)) return chunks chunks = split_transcript(transcript) labeled_chunks = [] for i, chunk in enumerate(chunks): prompt = f""" Label the following chunk of podcast transcript with speaker names. The host is Mark Rittman, and the guest is {guest_name}. Format the output as: Mark Rittman: [Speaker's words] {guest_name}: [Speaker's words] Use your understanding of conversation flow and context to accurately label each part of the dialogue. If you're unsure about a speaker, use your best judgment based on the content and style of speech. This is chunk {i+1} of {len(chunks)}. Maintain consistency with previous chunks if applicable. Transcript chunk: {chunk} """ response = client.chat.completions.create( model="gpt-4o", messages=[ {"role": "system", "content": "You are a transcript labeling assistant."}, {"role": "user", "content": prompt} ] ) labeled_chunks.append(response.choices[0].message.content) return "\n".join(labeled_chunks) ``` The transcription and all of the additional insights we’ve derived are written to a Google BigQuery table and dataset. ``` # Define the schema for your BigQuery table schema = [ bigquery.SchemaField("title", "STRING", mode="REQUIRED"), bigquery.SchemaField("link", "STRING"), bigquery.SchemaField("description", "STRING"), bigquery.SchemaField("published", "STRING"), bigquery.SchemaField("audio_url", "STRING"), bigquery.SchemaField("transcript", "STRING"), bigquery.SchemaField("classification", "STRING"), bigquery.SchemaField("summary_and_insights", "STRING"), # New field ] def ensure_table_exists(): dataset_ref = bq_client.dataset(dataset_name) table_ref = dataset_ref.table(table_name) try: bq_client.get_table(table_ref) except Exception: table = bigquery.Table(table_ref, schema=schema) table = bq_client.create_table(table) print(f"Created table {table.project}.{table.dataset_id}.{table.table_id}") def write_to_bigquery(episode): # Ensure the table exists ensure_table_exists() # Prepare the data for BigQuery row = { 'title': episode['title'], 'link': episode.get('link', ''), 'description': episode.get('description', '')[:1024], # Truncate description if it's too long 'published': episode.get('published', ''), 'audio_url': episode.get('audio_url', ''), 'transcript': episode.get('transcript', '')[:1048576], # Truncate transcript if it's too long (1MB limit) 'classification': episode.get('classification', ''), 'summary_and_insights': episode.get('summary_and_insights', '')[:1048576] # New field, also truncated if necessary } # Get the table reference table_ref = bq_client.dataset(dataset_name).table(table_name) # Load the data into BigQuery try: errors = bq_client.insert_rows_json(table_ref, [row]) if errors == []: print(f"Episode '{episode['title']}' successfully written to BigQuery") else: print(f"Errors occurred while writing episode '{episode['title']}' to BigQuery: {errors}") print(f"Problematic row: {row}") except google_exceptions.BadRequest as e: print(f"BadRequest error for episode '{episode['title']}': {e}") print(f"Problematic row: {row}") except Exception as e: print(f"Unexpected error for episode '{episode['title']}': {e}") print(f"Problematic row: {row}") # Add this function to check if rows were actually inserted def check_table_rows(): query = f""" SELECT COUNT(*) as row_count FROM `{bq_client.project}.{dataset_name}.{table_name}` """ query_job = bq_client.query(query) results = query_job.result() for row in results: print(f"Number of rows in the table: {row.row_count}") ``` Viewing the resulting BigQuery table we can see the episode transcripts along with all the other derived data and episode metadata. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2F398dc-630f94f0-ea34-48e2-854d-3e68f6b64602_1000x429.png%3Fssl%3D1) And finally the BigQuery dataset then becomes the data source for the [https://podcasts.rittmananalytics.com](https://podcasts.rittmananalytics.com/) web application, with the screenshot below showing an example episode transcript. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2Feb38a-cde61e66-da72-4091-8ece-3c02290d3c8b_1000x621.png%3Fssl%3D1) ### **INTERESTED? FIND OUT MORE!** Rittman Analytics is a boutique data analytics consultancy that helps ambitious, digital-native businesses scale-up their approach to data, analytics and [generative AI](https://www.rittmananalytics.com/services/genai). We’re authorised delivery partners for [Google Cloud](https://www.rittmananalytics.com/partners/looker) along with [Oracle](https://www.rittmananalytics.com/partners/oracle), [Segment](https://www.rittmananalytics.com/partners/segment), [Cube](https://www.rittmananalytics.com/partners/cube), [Dagster](https://www.rittmananalytics.com/partners/dagster), [Preset](https://www.rittmananalytics.com/partners/preset), [dbt Labs](https://www.rittmananalytics.com/partners/dbt-labs) and [Fivetran](https://www.rittmananalytics.com/partners/fivetran) and are experts at helping you design an analytics solution that’s right for your organisation’s needs, use-cases and budget and working with you and your data team to successfully implement it. If you’re looking for some help and assistance with your AI initiative or would just like to talk shop and share ideas and thoughts on what’s going on in your organisation and the wider data analytics world, [contact us now](https://calendly.com/markrittman/initial-discovery-call-with-mark-rittman?utm_source=medium.com&utm_medium=blog&utm_campaign=customer_segmentation) to organise a 100%-free, no-obligation call — we’d love to hear from you! --- # How Rittman Analytics Automates Project RAG Status Reporting using Vertex AI, DocumentAI, BigQuery & Looker Canonical: https://rittmananalytics.com/blog/2025/01/29/genai-looker-rag-status URL: https://rittmananalytics.com/blog/2025/01/29/genai-looker-rag-status Description: A common way to report on the progress of a project is through RAG (Red, Amber Green) status indicators for the project, at the overall level and by cost, schedule,… Author: Mark Rittman Published: 2025-01-29T17:00:45+00:00 Modified: 2025-01-29T17:00:45+00:00 --- ![How Rittman Analytics Automates Project RAG Status Reporting using Vertex AI, DocumentAI, BigQuery & Looker](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2F033e1-2b6b5931-448f-4565-97bb-a6fe006c0a00_1000x664.png%3Fssl%3D1) A common way to report on the progress of a project is through [RAG (Red, Amber Green) status indicators](https://en.wikipedia.org/wiki/Traffic_light_rating_system) for the project, at the overall level and by cost, schedule, scope, technology and data quality. We’re currently piloting a Looker dashboard of this type to provide RAG status updates to our clients along with key project metrics and progress towards objectives, as shown in the screenshot below. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2F033e1-2b6b5931-448f-4565-97bb-a6fe006c0a00_1000x664.png%3Fssl%3D1) What’s interesting and is the subject of this blog is how we’re using generative AI, in the form of Google’s Vertex AI’s [Gemini LLM](https://blog.google/technology/ai/google-gemini-ai/), to assess each project engagement using a wide-ranging set of structured, semi-structured and unstructured data sources such as meeting transcripts, statements of work and project documentation. #### **Why RAG Status Reporting Is Hard to Automate** Creating a Looker dashboard that reports-on the number of tickets closed, amount of budget spent and other project metrics is relatively-straightforward but the numbers only tell part of the story. What’s discussed in project meetings, in Slack messages and written in project documentation is the additional colour and context that’s used by a project manager when assigning RAG statuses to a project. And of course all of those RAG statuses are set by reference to the project objectives, expected benefits and agreed deliverables contained within the project Statement of Work, typically a PDF document that’s easy for a project manager to read but usually impenetrable to a dashboarding tool. To create our RAG status dashboard our approach was to source and centralize data from the following data sources and then use that data to prompt Vertex AI to classify the project engagement as either Red, Amber or Green RAG status: - Project metrics and KPI data from Harvest, Jira and Hubspot - Meeting transcripts from [Fathom.video](https://fathom.video/) - Messages from Slack - Project documentation from [Confluence Cloud](https://developer.atlassian.com/cloud/confluence/) - Statements of Work stored as PDFs and staged in Google Cloud Storage ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2Ffc652-616e955b-b820-401e-92e9-ec5a5d40dd90_1000x575.png%3Fssl%3D1) In this blog we’ll focus on the unstructured and semi-structured data sources that were new to our internal BigQuery warehouse and are good examples of how you can use [BQML functions](https://cloud.google.com/bigquery/docs/bqml-introduction) to parse and analyze these types of non-conventional warehouse sources. Let’s start with how we wrangled and added structure to our Fathom.video meeting transcripts, a source of project activity data that’s been key to this whole generative AI use-case. #### **Parsing & Structuring Fathom Meeting Transcripts** Like many organizations we’re big users of services such as Fathom.video that automatically transcribe meetings for you, and we’ve tried other service such as Fireflies.ai and Otter.ai in the past; crucially though for us, [Fathom’s integrates with the Zapier service](https://help.fathom.video/en/articles/450432) and enables us to set-up processing pipelines that can call other services, such as [Google Vertex A](https://zapier.com/apps/google-vertex-ai/integrations)I, and then [store the final output](https://zapier.com/apps/google-bigquery/integrations) in our Google BigQuery data warehouse. When a new meeting transcript is published, we run that transcript raw text through a number of processing steps in a Zapier pipeline, analyzing the transcript’s overall sentiment, summarising it and then classifying the meeting as either an internal meeting or one of two client meeting types. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2Fee30e-c7d1ff2c-730d-4ee5-aaa3-cfddd4780ff9_1000x984.png%3Fssl%3D1) Each step in the pipeline sends a prompt to Vertex AI’s Gemini Flash-1.5 LLM to ask it to analyze the sentiment of the whole transcript, summarise it and classify it as either a client or internal meeting, with the raw transcript and all of the LLM-generated additions then written to a table in our Google BigQuery warehouse. Taking just the raw transcript for now, we then process it using a set of dbt transformations as shown in the dbt Cloud lineage view below: ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2F07318-beddfd43-ef32-4352-b9d2-8a1714ddfb28_1000x251.png%3Fssl%3D1) Each raw transformation is structured in pairs of two lines, the first line containing the time, speaker name and company name in brackets and the following line containing the words that person said, as shown in the example below: ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2Ff33a1-2b2c5b5a-cecc-4037-86ce-da236a9d1cd9_1000x210.png%3Fssl%3D1) We then parse this semi-structured data using the dbt staging-layer model below, using BigQuery SQL to split the raw text field content by newlines, group those lines together in pairs and then join them together to return one row per transcript speaker line. ``` SplitTranscript AS ( -- Split the entire transcript into individual lines SELECT recording_url, -- Optional identifier for the transcript meeting_scheduled_start_time as meeting_start_ts, ARRAY_AGG(line) OVER (PARTITION BY recording_url ORDER BY line_number) AS transcript_lines FROM ( SELECT recording_url, meeting_scheduled_start_time, line_number, line FROM ( SELECT recording_url, meeting_scheduled_start_time, line_number, line FROM ( SELECT recording_url, meeting_scheduled_start_time, SPLIT(transcript_plaintext, '\n') AS lines -- Split the transcript by newlines FROM source ), UNNEST(lines) AS line WITH OFFSET AS line_number -- Create a line index ) ) ), GroupedTranscript AS ( SELECT recording_url, meeting_start_ts, ARRAY( SELECT STRUCT( transcript_lines[SAFE_OFFSET(i)] AS metadata_line, transcript_lines[SAFE_OFFSET(i + 1)] AS text_line ) FROM UNNEST(transcript_lines) AS line WITH OFFSET i WHERE MOD(i, 2) = 0 -- Pair lines: metadata (even indices) with text (odd indices) ) AS paired_lines FROM SplitTranscript ), ParsedTranscript AS ( SELECT recording_url, REGEXP_EXTRACT(metadata_line, r'(\d+:\d+:\d+)') AS time_string, TIMESTAMP_ADD( meeting_start_ts, INTERVAL TIME_DIFF(cast(REGEXP_EXTRACT(metadata_line, r'(\d+:\d+:\d+)') as time), TIME '00:00:00', SECOND) SECOND ) AS time, trim(REGEXP_EXTRACT(metadata_line, r' - ([^(]+)')) AS contact_name, trim(REGEXP_EXTRACT(metadata_line, r'\(([^)]+)\)')) AS company_name, text_line AS text FROM GroupedTranscript, UNNEST(paired_lines) ) SELECT time, recording_url, concat('fathom-',contact_name) as contact_id, trim(contact_name) as contact_name, concat('fathom-',company_name) as company_id, trim(company_name) as company_name, text FROM ParsedTranscript WHERE time IS NOT NULL AND text IS NOT NULL GROUP BY 1,2,3,4,5,6,7 ``` This parsing not-only splits the meeting transcript into one row per-line of each meeting attendees’ words but it also parses-out the speaker names and their company names into their own table columns, which we use later-on in the pipeline to join these meeting conversations back to our central warehouse “contacts” dimension. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2F12122-29b08bb0-0564-44aa-a51c-e1b3b57bb37c_1000x240.png%3Fssl%3D1) And now that we have the transcript in this more structured form, we can further process and aggregate the meeting contributions from each person through a series of dbt transformations as shown in the diagram below. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2Fcd1e7-88ba2c9b-0eca-469d-a519-f131c40b92c5_1000x509.png%3Fssl%3D1) This BigQuery-powered dbt pipeline creates for us: 1. Per meeting attendee, a summary of what points they made in the meeting (their “contribution”) alongside a summary of the meeting as a whole, to help us understand the viewpoint, sentiment and engagement in that meeting for each attendee 2. Using all of a meeting attendees’ contributions that we aggregate into a single JSON string we then use BigQuery’s [ML.GENERATE\_TEXT](https://cloud.google.com/bigquery/docs/generate-text) function to pass these contributions to an LLM and ask it to return a JSON-formatted list of “jobs-to-be-done” for that person. ``` meeting_contributions AS ( SELECT max(contact_fk) as contact_fk, max(company_fk) as company_fk, trim(contact_name) as contact_name, CONCAT( '[', STRING_AGG( TO_JSON_STRING( STRUCT( meeting_start_ts,meeting_title,meeting_contribution, contribution_sentiment_category,meeting_engagement_level ) ) ORDER BY meeting_start_ts DESC ), ']' ) AS all_contributions FROM meeting_contributions_deduped GROUP BY 3 ), jtdbs AS ( SELECT contact_fk, company_fk, trim(contact_name) as contact_name, REPLACE(REPLACE(REPLACE(ml_generate_text_llm_result, '```json', ''),'```',''),'\n','') AS jtbd_output FROM ML.GENERATE_TEXT( MODEL `analytics_ai.gemini_1_5_flash`, ( SELECT contact_fk, company_fk, trim(contact_name) as contact_name, CONCAT( 'You are a business consultant specialising in extracting actionable insights. Your role is to analyse the meeting contributions of ', contact_name, ' and extract the main Jobs to Be Done (JTBDs) for their involvement in a specific project. Your task involves: 1. Identifying the primary JTBDs from the given contributions. 2. Listing the JTBDs clearly and concisely. 3. Providing a high-level summary of the identified JTBDs. 4. Extracting the most meaningful keywords from the JTBD descriptions. Ensure that these keywords are concise, reflective of the main themes, and avoid generic or overly common words. Output the results in a JSON format suitable for parsing and use in Google BigQuery, using the following structure: { "contact_name": "Contact Name", "identified_jtbds": [ { "jtbd_id": 1, "description": "Description of the first JTBD" } ], "summary": "High-level summary of the identified JTBDs.", "keywords": ["keyword1", "keyword2", "keyword3"] } Follow this schema strictly to ensure compatibility with downstream processing systems. The contributions from this person were: ', meeting_contributions.all_contributions ) AS prompt FROM meeting_contributions ), STRUCT( 0.2 AS temperature, 4096 AS max_output_tokens, TRUE AS flatten_json_output ) ) ), parsed_jtbd AS ( SELECT contact_fk, company_fk, trim(contact_name) as contact_name, JSON_VALUE(jtbd_output, '$.summary') AS jtbd_summary, ARRAY( SELECT AS STRUCT CAST(JSON_VALUE(jtb, '$.jtbd_id') AS INT64) AS jtbd_id, JSON_VALUE(jtb, '$.description') AS description FROM UNNEST(JSON_EXTRACT_ARRAY(jtbd_output, '$.identified_jtbds')) AS jtb ) AS identified_jtbds, ARRAY( SELECT keyword FROM UNNEST(SPLIT(REPLACE(REPLACE(REPLACE(JSON_QUERY(jtbd_output, '$.keywords'), '[', ''), ']', ''),'"',''), ',')) AS keyword WHERE keyword IS NOT NULL ) AS keywords FROM jtdbs ) SELECT * FROM parsed_jtbd ``` The JSON-formatted results of this prompt response are then transform into two BigQuery nested sets of columns, one containing their jobs-to-be-done and the other a list of keywords from those jobs as shown in the example output below. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2F1747a-c4214f8d-ebe7-4953-86b0-aa598f8d7c6b_1000x389.png%3Fssl%3D1) You can read about other use-cases we’ve developed for BigQuery’s ML.GENERATE\_TEXT function in our blogs on [Automating your Contacts List Segmentation using Google BigQuery, Vertex AI and the ML.GENERATE\_TEXT function](https://www.rittmananalytics.com/blog/2024/1/14/automate-your-contacts-list-segmentation-using-google-bigquery-vertex-ai-and-the-mlgeneratetext-function) and [Automating Insights and Dashboard Report-Level Narratives using Vertex AI Gemini 1.5-Flash, Looker Studio and Google BigQuery BQML](https://www.rittmananalytics.com/blog/2024/6/17/automated-insights-gemini-1-5-flash). #### **Confluence Storage Format Parsing using Gen AI** Another key source of data for assessing the progress of a project is the documentation we create as meeting notes and the final engagement deliverables. [Fivetran have a Confluence Cloud connector](https://fivetran.com/docs/connectors/applications/confluence) that extracts into BigQuery all of the page and comments data we’re looking for, but the raw form of the content it brings across is in a weird [XHTML Confluence Storage Format](https://confluence.atlassian.com/doc/confluence-storage-format-790796544.html) that isn’t easily parsed by BigQuery’s JSON query functional; also all of the content is versioned and so needs filtering to return the most recent version ``` SELECT concat(concat('{{var('stg_confluence_docs_id-prefix')}}',s.name),'-',p.id,'-',p.version_number) as page_key, concat('{{var('stg_confluence_docs_id-prefix')}}',s.name) as company_id, s.name as space_name, p.id AS page_id, p.title as page_title, p.version_number as page_version_number, JSON_VALUE(v.page_body, '$.atlas_doc_format.value') as page_body , v.created_at as page_created_at_ts, MAX(v.created_at) OVER (PARTITION BY p.id ORDER BY v.created_at RANGE BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING) = v.created_at AS is_latest_version FROM ... ``` To parse this XHTML content we also used BigQuery’s ML.GENERATE\_TEXT function, this time asking the Vertex AI LLM to first parse the XHTML-formatted data from Confluence and then summarise it, with those summaries then used later-on as parts of the input into the RAG status generation dbt model. ``` SELECT * except(prompt,ml_generate_text_llm_result,ml_generate_text_rai_result,ml_generate_text_status), JSON_VALUE(REPLACE(REPLACE(REPLACE(ml_generate_text_llm_result, '```json', ''),'```',''),'\n',''),'$.page_contents') page_contents, JSON_VALUE(REPLACE(REPLACE(REPLACE(ml_generate_text_llm_result, '```json', ''),'```',''),'\n',''),'$.page_summary') page_summary FROM ML.GENERATE_TEXT( MODEL `{{var('stg_confluence_docs_fivetran_llm_name')}}`, ( SELECT page_key, company_id, space_name, page_id, page_title, page_version_number, page_created_at_ts, CONCAT('For the page entited "', coalesce(page_title,''), '" in the confluence space named "',coalesce('space_name',''), '" Please extract the exact content of the page (with html tags stripped) and a summary of those contents and return it in JSON in this format : {\"page_contents\": \"\", \"page_summary\": \"\"} : ',j.page_body) AS prompt FROM confluence_json j), STRUCT( {{var('stg_confluence_docs_fivetran_llm_temperature')}} AS temperature, {{var('stg_confluence_docs_fivetran_llm_max_output_tokens')}} AS max_output_tokens, TRUE AS flatten_json_output)) ``` If you’re wondering, the various REPLACE() functions around the output are to get rid of the extraneous “\`json and linebreaks added by the LLM that would otherwise stop BigQuery’s JSON\_VALUE function from working. #### **Statement of Work PDF Parsing using DocumentAI** To usefully assess the current status and progress of a project for a client, we really need to know what the original requirements, business objectives and agreed deliverables were for that project, information that can be found in a project-specific document called a “Statement of Work”, a free-form loosely-structured document stored as a PDF document. Our strategy to access this statement of work data is to load it as embeddings into a vector storage table and then have ML.GENERATE\_TEXT access it as a RAG (Retrieval Augmented Generation, another type of “RAG”) semantic search source, using two more BQML functions, [ML.PROCESS\_DOCUMENT](https://cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-process-document) and [ML.GENERATE\_EMBEDDING](https://cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-generate-embedding). ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2F46d64-f929fbf8-4019-4fa4-a8bc-66e6c25ab9c5_1000x281.png%3Fssl%3D1) The steps to implement this pipeline are implemented by two Python notebooks, one that is run once-only and sets the pipeline up ([code example here](https://github.com/rittmananalytics/rag_status_dashboard_example_code/blob/main/setup_pdf_ingestion.ipynb)) and a second that loops through all of the PDFs in Google Cloud Storage and, for all that have not yet been ingested, runs through the following steps ([code example here](https://github.com/rittmananalytics/rag_status_dashboard_example_code/blob/main/run_pdf_ingestion.ipynb)) 1. Define a BigQuery Object Table over each SoW PDF file using the [CREATE EXTERNAL TABLE](https://cloud.google.com/bigquery/docs/object-table-introduction) DDL command ``` CREATE OR REPLACE EXTERNAL TABLE `{DATASET_ID}.{table_name}` WITH CONNECTION `{CONNECTION_NAME}` OPTIONS ( uris = ['gs://{BUCKET_NAME}/{blob.name}'], object_metadata = 'DIRECTORY' ); ``` 2\. Parse, chunk and create embeddings for the PDF data using the BQML [ML.PROCESS\_DOCUMENT](https://cloud.google.com/blog/products/data-analytics/bigquery-and-document-ai-layout-parser-for-document-preprocessing) function and the DocumentAI [Layout Parser processor](https://cloud.google.com/document-ai/docs/layout-parse-chunk) (note that if, like us, you’re based in the UK and use the europe-west2 Google Cloud Platform region this feature is only available in the EU and US multi-regions, and so you’ll have to parse the documents in the EU region and then copy the final output table over to europe-west2 at the end to bring this data into europe-west2) ``` CREATE OR REPLACE TABLE {DATASET_ID}.{result_table_name} AS ( SELECT * FROM ML.PROCESS_DOCUMENT( MODEL {DATASET_ID}.layout_parser, TABLE {DATASET_ID}.{table_name}, PROCESS_OPTIONS => (JSON '{{"layout_config": {{"chunking_config": {{"chunk_size": 250}}}}}}') ) ); CREATE OR REPLACE TABLE {DATASET_ID}.{result_parsed_table_name} AS ( SELECT uri, JSON_EXTRACT_SCALAR(json , ' .content') AS content, JSON_EXTRACT_SCALAR(json , ' .pageSpan.pageStart') AS page_span_start, JSON_EXTRACT_SCALAR(json , '$.pageSpan.pageEnd') AS page_span_end FROM {DATASET_ID}.{result_table_name}, UNNEST(JSON_EXTRACT_ARRAY(ml_process_document_result.chunkedDocument.chunks, '$')) json ); ``` 3\. Generate vector embeddings for each of those parsed results tables using the ML.GENERATE\_EMBEDDINGS function ``` CREATE OR REPLACE TABLE {DATASET_ID}.{embeddings_table_name} AS SELECT * FROM ML.GENERATE_EMBEDDING( MODEL {DATASET_ID}.embedding_model, TABLE {DATASET_ID}.{result_parsed_table_name} ); ``` 4\. And Finally, use the ML.GENERATE\_TEXT function to create a prompt that uses the vector table as your RAG search source and outputs, in a structured form, the details in the engagement Statement of Work. ``` select * from (with objectives as ( SELECT {deal_id} AS deal_id, ml_generate_text_llm_result AS objectives_json, NULL AS engagement_summary, NULL AS engagement_deliverables FROM ML.GENERATE_TEXT( MODEL `{DATASET_ID}.gemini_flash`, ( SELECT CONCAT( 'What are the (up to 10) objectives, expected business benefits and measurement metrics for this project? Be concise, add no preamble and respond in JSON format using output fields "objective_number", "objective_details", "expected_benefit","objective_metric" using the following context:', STRING_AGG(FORMAT("context: %s and reference: %s", base.content, base.uri), ',')) AS prompt FROM VECTOR_SEARCH( TABLE `{DATASET_ID}.{deal_id}_embeddings`, 'ml_generate_embedding_result', ( SELECT ml_generate_embedding_result, content AS query FROM ML.GENERATE_EMBEDDING( MODEL `{DATASET_ID}.embedding_model`, ( SELECT 'What are the objectives of this project? Be concise and use the following context' AS content ) ) ), TOP_K => 10, OPTIONS => '{{"fraction_lists_to_search": 0.01}}' ) ), STRUCT(2048 AS max_output_tokens, TRUE AS flatten_json_output) )), engagement_summary as ( -- Engagement summary for deal_id {deal_id} SELECT {deal_id} AS deal_id, NULL AS objectives_json, ml_generate_text_llm_result AS engagement_summary, NULL AS engagement_deliverables FROM ML.GENERATE_TEXT( MODEL `{DATASET_ID}.gemini_flash`, ( SELECT CONCAT( 'Summarise the background, business requirements, solution and assumptions for this project. Be concise and respond in JSON format using output fields "Background", "Requirements","Solution", "Assumptions", add no preamble and be factual:', STRING_AGG(FORMAT("context: %s and reference: %s", base.content, base.uri), ',')) AS prompt, FROM VECTOR_SEARCH( TABLE `{DATASET_ID}.{deal_id}_embeddings`, 'ml_generate_embedding_result', ( SELECT ml_generate_embedding_result, content AS query FROM ML.GENERATE_EMBEDDING( MODEL `{DATASET_ID}.embedding_model`, ( SELECT 'Summarise the background, requirements and solution for this project' AS content ) ) ), TOP_K => 10, OPTIONS => '{{"fraction_lists_to_search": 0.01}}' ) ), STRUCT(2048 AS max_output_tokens, TRUE AS flatten_json_output) )), engagement_deliverables as ( SELECT {deal_id} AS deal_id, NULL AS objectives_json, NULL AS engagement_summary, ml_generate_text_llm_result AS engagement_deliverables FROM ML.GENERATE_TEXT( MODEL `{DATASET_ID}.gemini_flash`, ( SELECT CONCAT( 'What are the (up to 10) contractual deliverables listed for this project? Be concise, add no preamble and respond in JSON format using output fields "deliverable_number", "deliverable_details", "deliverable_format", "acceptance_criteria" using the following context:', STRING_AGG(FORMAT("context: %s and reference: %s", base.content, base.uri), ',')) AS prompt FROM VECTOR_SEARCH( TABLE `{DATASET_ID}.{deal_id}_embeddings`, 'ml_generate_embedding_result', ( SELECT ml_generate_embedding_result, content AS query FROM ML.GENERATE_EMBEDDING( MODEL `{DATASET_ID}.embedding_model`, ( SELECT 'What are the (up to 10) contractual deliverables listed for this project? Be concise and use the following context' AS content ) ) ), TOP_K => 10, OPTIONS => '{{"fraction_lists_to_search": 0.01}}' ) ), STRUCT(2048 AS max_output_tokens, TRUE AS flatten_json_output) )) SELECT o.deal_id, JSON_VALUE(REPLACE(REPLACE(s.engagement_summary,'```json',''),'```',''), " .Requirements") AS requirements, JSON_VALUE(REPLACE(REPLACE(s.engagement_summary,'```json',''),'```',''), " .objective_number') AS objective_number, JSON_EXTRACT_SCALAR(item, ' .expected_benefit') AS expected_benefit, JSON_EXTRACT_SCALAR(item, ' .deliverable_number') AS deliverable_number, JSON_EXTRACT_SCALAR(item, ' .deliverable_format') AS deliverable_format, JSON_EXTRACT_SCALAR(item, '$.acceptance_criteria') AS acceptance_criteria FROM UNNEST(JSON_EXTRACT_ARRAY(REPLACE(REPLACE(d.engagement_deliverables,'```json',''),'```',''))) AS item ) AS deliverables, REPLACE(REPLACE(o.objectives_json,'```json',''),'```','') as objectives_json, REPLACE(REPLACE(d.engagement_deliverables,'```json',''),'```','') as deliverables_json FROM objectives o LEFT JOIN engagement_summary s ON o.deal_id = s.deal_id LEFT JOIN engagement_deliverables d ON o.deal_id = d.deal_id) """) ``` Looking now at the results of this processing pipeline you can see one row per engagement SoW PDF with the background, requirements, solution and then nested column sets for the objectives (including expected business benefit and measurement metric) and the contractual deliverables (with acceptance criteria). ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2F90a05-07cc10f1-394d-4077-8060-7c62e73605ea_1000x493.png%3Fssl%3D1) In-total, including the pipelines to bring-in Slack messaging data and the various project KPI data sources such as Jira, Harvest and Hubspot, the complete processing pipeline for assessing the project engagement’s RAG status is shown in the diagram below. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2F7512b-597091a4-d5a2-45ee-80ca-ae7ab6f7afce_1000x881.png%3Fssl%3D1) #### **Using ML.GENERATE\_TEXT to Classify RAG Status** The final step in this process is to take all of the data gathered from the various project sources, concatenate the multi-row datasets they provide into JSON strings suitable for sending as part of an LLM prompt and then use them to create a series of prompts to the Gemini 1.5-FLASH LLM using BigQuery’s ML.GENERATE\_TEXT function. The full SQL query can be accessed as [sample code here](https://github.com/rittmananalytics/rag_status_dashboard_example_code/blob/main/rag_status_bqml_query.sql), but the key part of the command is where we concatenate all of these data sources into a prompt we send to the LLM, shown below: ``` FROM ML.GENERATE_TEXT( MODEL `analytics_ai.gemini_1_5_flash`, ( SELECT mc.company_name, mc.deal_name, mc.deal_description, mc.project_name, mc.engagement_month, CONCAT( 'Provide a RAG status report for a Rittman Analytics client project in the following JSON format, where each field is filled with a brief, clear, and specific explanation of the project status in the context of the engagement background and requirements, the set of business objectives and contractual deliverables, the planned overall solution and the project documents published, meeting transcripts, slack messages and other project output and evidence. The possible statuses are RED, AMBER, or GREEN and the rationales should succinctly explain why each status was assigned (citing evidence), taking care to consider whether the engagement and project are for Discovery, Planning or Design and where the calculation of RAG status should relate to Rittman Analytics scope, financials, resourcing, schedule, technology OR the engagement is a project implementation where we are then responsible for the clients current technology, data quality, project scope, resourcing and schedule and for the project financials). In addition, provide a timeline of the significant (not all) activities (excluding daily standups and exchanges of pleasantries) and outputs of the engagement this month to accompany the RAG statuses. Use the format {\"overall_status\": \"\", \"overall_rationale\": \"\", \"scope_status\": \"\", \"scope_rationale\": \"\", \"financials_status\": \"\", \"financials_rationale\": \"\", \"schedule_status\": \"\", \"schedule_rationale\": \"\", \"data_quality_qa_status\": \"\", \"data_quality_qa_rationale\": \"\", \"technology_status\": \"\", \"technology_rationale\": \"\", \"resourcing_status\": \"\", \"resourcing_rationale\": \"\", \"month_timeline\": \"= DATE(p.project_delivery_start_ts) AND date <= DATE(p.project_delivery_end_ts), date, NULL)) AS business_days_in_month FROM date_series d JOIN projects_by_engagement p ON d.timesheet_project_pk = p.timesheet_project_pk LEFT JOIN UNNEST(GENERATE_DATE_ARRAY(DATE(d.month_start), DATE(d.month_end), INTERVAL 1 DAY)) AS date GROUP BY 1, 2, 3, 4, 5, 6, 7, 8, 9 ) ``` 3\. Next we use those counts of business days per month to allocate revenue to each of those months, like this: ``` WITH projects_by_engagement_monthly AS ( SELECT company_name, timesheet_project_pk, project_name, engagement_code, month_start, project_delivery_start_ts, project_delivery_end_ts, business_days_in_month, total_recognized_revenue_per_working_day, ROUND(business_days_in_month * total_recognized_revenue_per_working_day, 2) AS monthly_recognized_revenue FROM monthly_business_days ) ``` 4\. Finally, we use a SUM() OVER () window function to calculate the remaining revenue for each engagement as the months progress and then subtract the cumulative revenue already billed from the total engagement value to give us, per engagement per month, the total remaining unbilled revenue. ``` WITH engagement_monthly_revenue AS ( SELECT company_name, engagement_code, month_start, SUM(monthly_recognized_revenue) AS monthly_recognized_revenue FROM projects_by_engagement_monthly GROUP BY 1, 2, 3 ) SELECT {{ dbt_utils.surrogate_key(['month_start','engagement_code']) }} AS project_engagement_monthly_revenue_pk, company_name, engagement_code, month_start, SUM(monthly_recognized_revenue) OVER (PARTITION BY engagement_code) - COALESCE(SUM(monthly_recognized_revenue) OVER (PARTITION BY engagement_code ORDER BY month_start ROWS BETWEEN UNBOUNDED PRECEDING AND 1 PRECEDING), 0) AS total_engagement_revenue_remaining FROM engagement_monthly_revenue ``` From this SQL analysis we then create three Looker data visualizations: 1. **Booked Revenue Pipeline Chart** that visualizes the total remaining booked revenue by month, giving an at-a-glance view of how much confirmed work our business has in the pipeline for future months. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2F3cb90-1bb07151-7292-472a-a166-03f1eb896d12_1000x352.png%3Fssl%3D1) 2\. **Delivered vs. Booked Pipeline Chart** and **Revenue Months in Pipeline Chart** compare the booked revenue against delivered revenue (i.e., the revenue that has already been billed), telling us how fast we are getting-through our backlog of booked projects and showing how long the company’s backlog of work can sustain future revenue. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2F29835-317ca38c-88a5-4540-bf01-5ea009c03380_1000x396.png%3Fssl%3D1) #### Doing the Same for our Sales Pipeline using Stitch and Fivetran While the **booked revenue pipeline** provides a view of our confirmed (booked) project work yet to be delivered, we also need to evaluate our sales pipeline — the potential revenue still in negotiation. Analyzing this pipeline is critical for understanding future business opportunities and ensuring that our long-term revenue stream remains strong. However, an approach where we rely solely on traditional sales pipeline metrics such as value of new and closed deals per month and tracking all open deals by their current pipeline stage won’t be sufficient because: 1. A deal in the early stages of the sales process has a much lower likelihood of closing than a deal in the final negotiation stage. Simply tracking the total value of all deals in the pipeline ignores these probabilities and can lead to inflated expectations about future revenue 2. The value of deals in the sales pipeline can shift over time as deals move between stages. Relying on static snapshots of deal value, without accounting for historical progress, does not provide insight into how much potential revenue was in the pipeline at specific points in time. 3. Without applying stage-specific weightings to deals, we can’t realistically assess the potential revenue in the pipeline. A deal in the initial meeting stage has a much lower probability of converting into closed revenue than a deal in the negotiation stage and if all deals are treated equally this can lead to an overestimation of our potential future revenue. Instead we track sales opportunities over time, seeing the total pipeline value at different points in history, including both won, lost, and open opportunities; and we apply stage-specific weights to open deals, adjusting the potential revenue to reflect the historical success rate at each stage. For example, deals in the initial meeting stage historically have a 30% chance of closing while those in the proposal and negotiation stages have 45% and 60% chance of closing, based on our historical deal data from Hubspot. To generate our sales pipeline over time we use data extracted from Hubspot using both Fivetran and Stitch and then process it using dbt and Google BigQuery like this: 1. First we extract the core deal information, such as the deal’s ID, creation date, and name. We use Fivetran to pull current deal information, including the deal’s stage which we then categorize into a smaller set of stage categories (e.g., Initial Meeting, Needs Analysis, Negotiation) based on their position in the pipeline. ``` CASE WHEN p.display_order <= 2 THEN '1: Initial Meeting' WHEN p.display_order BETWEEN 2 AND 3 THEN '2: Needs analysis and proposal' WHEN p.display_order BETWEEN 4 AND 7 THEN '3: Negotiation and commitment' WHEN p.display_order BETWEEN 8 AND 9 THEN '4: Deal close' WHEN p.display_order = 10 THEN '0: Lost' END AS deal_stage_category ``` 2\. Then we make use of deal data extracted via Stitch, another data replication service, which snapshots the pipeline status of each Hubspot deal each time it runs which we then use to calculate the stage of the deal at each month-end. ``` DATE_TRUNC(DATE(entered_pipeline_stage_ts), MONTH) AS month_name ``` This allows us to construct a point-in-time view of the sales pipeline, ensuring that we can track the historical position of each deal at month-end as opposed to the current status, today, of that deal. You could equally achieve the same outcome by using dbt’s snapshot feature designed for slowly-changing dimensions or use the from-and-to dates for deal stages that Hubspot provides for deal records. 3\. An essential part of analyzing the pipeline is correctly handling closed deals (either won or lost). Deals that have been closed should not appear in future months as part of the open pipeline, but they must be accounted for in the month they closed. To handle this, the SQL query flags when a deal has reached the “Deal Close” or “Lost” stage. We then sum these closing events and calculate a cumulative sum of those closing event totals to ensure that once a deal is closed, it doesn’t reappear in the pipeline in subsequent months. ``` SUM(is_closing_events) OVER (PARTITION BY deal_id ORDER BY deal_month_ts RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS total_closing_events ``` 4\. Finally we calculate the deal value at the end of each month using the LAST\_VALUE window function, so that we have the most up-to-date value for a deal when the value potentially goes up-or-down during the month ``` LAST_VALUE(deal_amount_gbp) OVER (PARTITION BY deal_id, month_name ORDER BY entered_pipeline_stage_ts) AS deal_amount_gbp ``` We then use Looker to visualize our sales pipeline by stage over time using two visualizations: 1. **Pipeline by Stage Over Time**: This chart categorizes the pipeline by stage over time, giving a historical view of the pipeline mix at the end of each month and showing whether the sales pipeline has been predominantly filled with early-stage deals or if more deals are progressing to later stages, such as negotiation and deal close. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2F02760-b8ab3544-8219-47a1-a766-616c3ceb296c_1000x504.png%3Fssl%3D1) 2\. **Deal Pipeline Amount vs. Weighted** calculates a weighted deal value for deals in various stages based on their likelihood of closing and helps in forecasting more accurately, showing a realistic value of potential revenue based on the probability of deals closing. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2Fbd74f-13954e1c-008e-4fd9-929d-35dc6d428f60_1000x505.png%3Fssl%3D1) #### Calculating and Visualizing our Complete Revenue Pipeline Over-Time After analyzing both the **booked revenue pipeline** and the **sales pipeline** independently, the next step is to combine them in two final Looker visualizatios to get a holistic view of the total revenue pipeline, helping us understand not just the secured work that has been won but also the potential future revenue from deals still in the sales process. 1. **Total Pipeline by Month** merges the data from both the booked revenue pipeline and the sales pipeline (using weighted values for open deals), showing us the total potential revenue for each month made up of both booked revenue and weighted sales opportunities, providing a clear breakdown of how much of the pipeline is secured vs. still being negotiated. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2Fb924e-6fc3465f-102c-49b2-b827-7e057cfbd2c8_1000x450.png%3Fssl%3D1) 2\. **Pipeline Booked % vs. Total Pipeline** is critical for understanding the balance between booked and unbooked work and the level of risk in the current pipeline. It reveals situations where the total pipeline may appear large, but only a small percentage is actually confirmed which may or may not actually close. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2F229c0-5686735d-1573-4582-8bd2-e5edfe0c8cb6_1000x509.png%3Fssl%3D1) # **Interested? Find Out More!** Rittman Analytics is a boutique data analytics consultancy that helps ambitious, digital-native businesses scale-up and upgrade their sales, marketing and financial analytics capabilities, just like we’ve done for ourselves in this blog article. We’re authorised delivery partners for [Google Cloud](https://www.rittmananalytics.com/partners/looker) along with [Oracle](https://www.rittmananalytics.com/partners/oracle), [Segment](https://www.rittmananalytics.com/partners/segment), [Cube](https://www.rittmananalytics.com/partners/cube), [Dagster](https://www.rittmananalytics.com/partners/dagster), [Preset](https://www.rittmananalytics.com/partners/preset), [dbt Labs](https://www.rittmananalytics.com/partners/dbt-labs) and [Fivetran](https://www.rittmananalytics.com/partners/fivetran) and are experts at helping you build financial analytics and scalable, cloud-hosted data stacks that are designed for your organisation’s needs, use-cases and budget. If you’re looking for some help and assistance with your sales pipeline reporting or would just like to talk shop and share ideas and thoughts on what’s going on in your organisation and the wider data analytics world, [contact us now](https://calendly.com/markrittman/initial-discovery-call-with-mark-rittman?utm_source=medium.com&utm_medium=blog&utm_campaign=customer_segmentation) to organise a 100%-free, no-obligation call — we’d love to hear from you! --- # How Rittman Analytics Automates our Profit & Loss Reporting and Commentary using VertexAI Gemini 1.5-FLASH and Google BigQuery Canonical: https://rittmananalytics.com/blog/2024/09/17/genai-profit-and-loss-report-analysis URL: https://rittmananalytics.com/blog/2024/09/17/genai-profit-and-loss-report-analysis Description: A Profit & Loss Report (or “Income Statement” in some countries) is a financial report, typically produced monthly, quarterly or annually, that sets out the income, direct and overheads costs… Author: Mark Rittman Published: 2024-09-17T10:23:05+00:00 Modified: 2024-09-17T10:23:05+00:00 --- ![How Rittman Analytics Automates our Profit & Loss Reporting and Commentary using VertexAI Gemini 1.5-FLASH and Google BigQuery](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2Fe7ee5-a0d8645d-a5b0-4d4c-b0db-2704734ee26a_1000x816.png%3Fssl%3D1) A Profit & Loss Report (or “Income Statement” in some countries) is a financial report, typically produced monthly, quarterly or annually, that sets out the income, direct and overheads costs and the final net profit for a business. If, like us, you use Xero to run your company accounts then your typical default Profit & Loss report looks like the one below, based on the Demo Company data available in Xero. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2Fe7ee5-a0d8645d-a5b0-4d4c-b0db-2704734ee26a_1000x816.png%3Fssl%3D1) It’s great to see your sales, gross profit, operating profit and what’s left after taxation on one page but as I said in our previous blog on [Automating Insights and Dashboard Report-Level Narratives](https://blog.rittmananalytics.com/automating-insights-and-dashboard-report-level-narratives-using-vertex-ai-gemini-1-5-flash-e06fdbfc93ce), producing the report is just the first stage in a data-driven organization; you have to then understand what the report is telling you and then work-out what actions need to be taken with this new information. In the case of a Profit & Loss report such as this one, I’d want to understand the answer to questions such as: - How does this month’s revenue, gross margin (the percentage of revenue left after direct costs are deducted) and net margin (the same but after overheads are deducted) compare to last month? And for quarter-to-date how do these compare against the last quarter-to-date? Year-to-date? And so on - For account categories such as Administrative Costs (aka “overheads”) which account groups are trending-up over the past few months, something I’d want to keep a handle on if elements of our back-office costs keep rising month-on-month - Similarly, if spend on a category or account group has risen suddenly this month, or if sales have jumped unexpectedly, what individual transactions were the root-cause of these changes? - If I have my budget figures also stored in Xero, how does this months, quarter’s and year-to-date numbers compare to plan and are we on-track to hit our overall targets? And then, once I have this information, what does then mean for our management plans for the coming months? Do we need to look at our costs and bring them back in-line with our original plan for the year? Are sales an issue and do we need to work on building our pipeline? In this blog we’ll look at how generative AI, in the form of Vertex AI’s Gemini 1.5-FLASH LLM, can be used to automate the analysis of this multi-level financial dataset to produce insightful and actionable report commentary such as the examples shown below. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2Ffe521-c9493fda-f12b-409f-970f-3bf03635e454_1000x890.png%3Fssl%3D1) In our previous article on [using gen AI to automate the production of report narratives](https://www.rittmananalytics.com/blog/2024/6/17/automated-insights-gemini-1-5-flash) we used a relative simple example where we asked an LLM to analyze a table of monthly KPI values and asked it to comment on this month’s performance compared to last months’ and quarter-to-date, and gave it some basic guidance on how profit was made-up of revenue minus costs and so forth. In the case of analyzing a profit and loss report though you need to be more structured in how you perform your calculations and approach the process of analyzing the result of those calculations and investigating root cause. Mirroring the thought process outlined earlier where we start at account category level and calculate all the period-on-period variances, then go down to account group level and finally identify the individual transactions that led to those variances and anomalies, we created a data pipeline that starts with the [dbt transformations that Fivetran can run](https://github.com/fivetran/dbt_xero) when replicating your Xero data into your data warehouse and created a [BigQuery Python notebook](https://github.com/rittmananalytics/Automated_Profit-Loss_Report_Analyzer_VertexAI/blob/main/Xero_Profit_and_Loss_Report_Analyzer.ipynb) that performed those same analysis steps, which we’ve made available to download along with a readme file from [our Github account](https://github.com/rittmananalytics/Automated_Profit-Loss_Report_Analyzer_VertexAI/tree/main). ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2F0c5dc-85bbb0fe-3ef2-46ec-8522-f2de7ce4c417_1000x451.png%3Fssl%3D1) Looking at the output from this stage of processing in that BigQuery notebook shows you enhanced P&L report dataset that we’ll pass to Vertex AI the following step to get our report commentary. The notebook then performs all of the trend analysis, root-cause identification and other complex financial calculations that need to be done in a specific, controlled and repeatable way using Python Pandas dataframe calculations, like this example that identifies significant transactions contributing to a variance in month-on-month account-level spend: ``` def get_significant_transactions(transactions_data, account_group, date, threshold=0.1, months_to_check=6): current_month = transactions_data[(transactions_data['account_report_group'] == account_group) & (transactions_data['date_month'] == date)] previous_months = transactions_data[(transactions_data['account_report_group'] == account_group) & (transactions_data['date_month'] > date - pd.DateOffset(months=months_to_check)) & (transactions_data['date_month'] < date)] total_amount = current_month['journal_net_amount'].sum() significant_transactions = current_month[abs(current_month['journal_net_amount']) > abs(total_amount * threshold)] # Identify and remove cancelling transactions trans_list = significant_transactions.to_dict('records') cancelling_pairs = identify_cancelling_transactions(trans_list) indices_to_remove = set([i for pair in cancelling_pairs for i in pair]) filtered_transactions = [trans for i, trans in enumerate(trans_list) if i not in indices_to_remove] # Add context for each transaction for transaction in filtered_transactions: similar_transactions = previous_months[ (previous_months['account_name'] == transaction['account_name']) & (previous_months['description'].str.contains(transaction['description'].split('-')[0], regex=False, na=False)) ] if not similar_transactions.empty: transaction['is_recurring'] = True transaction['previous_occurrences'] = len(similar_transactions) transaction['average_previous_amount'] = similar_transactions['journal_net_amount'].mean() transaction['change'] = transaction['journal_net_amount'] - transaction['average_previous_amount'] transaction['is_new'] = False else: transaction['is_recurring'] = False transaction['previous_occurrences'] = 0 transaction['average_previous_amount'] = 0 transaction['change'] = transaction['journal_net_amount'] transaction['is_new'] = True # Categorize the transaction transaction['category'] = categorize_transaction(transaction['description'], transaction['account_name']) return filtered_transactions ``` The way that we then structured the prompt that was then sent to the Gemini 1.5-FLASH model was key to the quality of the report analysis and commentary: ``` Analyze the following Profit and Loss data for {month} and provide a management report: {financial_data} Account Group Details (including significant transactions): {account_group_details} Cost Change Analysis: {cost_change_analysis} Do not use markdown formatting, * or #. All financial amounts are in GBP (£) **Focus on:** 1. Analyze Revenue, Cost of Delivery, and Overheads, highlighting significant account groups and transactions that have influenced these categories. 2. Pay special attention to the significant transactions listed for each account group. Explain how these transactions have impacted the overall performance of their respective categories. 3. Explain how changes in these transactional categories, including the effect of significant transactions, have impacted Gross Margin, Net Profit, and Retained Earnings. 4. Compare against previous month actual, calling out significant increases/decreases in the numbers, and relate these to any anomalous transactions where relevant. 5. Analyze YTD vs. LYTD performance, highlighting areas of over- and under-performance, considering the impact of significant transactions on these trends. 6. Ignore any double-entry but cancelling transactions, such as a negative expense followed by a positive expense of the same amount. These are typically corrections of administrative errors and should not be considered in the analysis. 7. When analyzing cost changes, focus on the actual causes of increases or decreases as shown in the Cost Change Analysis. Do not attribute changes to new hires or costs unless explicitly stated in the analysis. 8. Ensure all financial calculations are consistent. When discussing costs, always treat them as positive values for calculation purposes, even if they are recorded as negative values in the accounting system. 9. When describing changes in costs or revenues, ensure that the direction of change (increase/decrease) matches the sign of the percentage change. For example, if a cost decreased, the percentage change should be described as a decrease (negative percentage), not an increase. 10. Do not speculate about the reason any changes or transactions you see, and never state any person (e.g. staff) names when discussing salaries, bonuses or dividends: When analyzing significant transactions, provide context for each transaction: 1. Indicate whether it's a new transaction that wasn't present in the previous months. 2. If it's not new, specify how much it has changed compared to its average in previous months. 3. Explain the potential reasons for new transactions or significant changes in existing ones. 4. For recurring transactions, avoid describing them as "new" and instead focus on any changes in their amounts. When discussing changes in categories or specific expenses: 1. Always include the current amount, previous amount, and the percentage change. 2. For new expenses or categories, clearly state that they are new and provide their amounts. 3. Use the following format for discussing changes: "[Category] [increased/decreased] to [current amount] in [current month], compared to [previous amount] in [previous month], a change of [percentage]%." 4. When mentioning new expenses, use the format: "A new expense for [category] was introduced, amounting to [amount]." 5. For recurring expenses that have changed, use the format: "The expense for [category] [increased/decreased] to [current amount], compared to its previous average of [average amount], a change of [percentage]%." ** Output your analysis in a concise and informative manner, suitable for senior management and executives. Format your response using the following HTML structure:** ** Use the Economist Style Guide as your writing style, with bold tags on important elements such as account group names and anomalous transaction details. Do not use markdown formatting or "**" in your responses.

📊 Executive Summary

  • 📈 [Key summary point about overall financial performance in the context of change vs. last month]
  • 🔮 [Key summary point about future outlook or recommendations, considering observed anomalies]

💰 Revenue

  • [Key point about revenue]
  • [Explanation of significant variations in revenue account groups, including impact of notable transactions]

💼 Direct Costs

  • [Key point about direct costs]
  • [Explanation of the top one or two significant variations in direct costs account groups, including impact of the one most notable transaction in each delivery group. For each significant transaction, specify whether it's new or how it has changed from its average in previous months.]
  • [Analysis of the one most significant cost change, focusing on the actual causes of increases or decreases as shown in the Cost Change Analysis]

🏭 Gross Margin

  • [Key point about gross margin]
  • [Explanation of how changes in Revenue and Cost of Delivery, including significant transactions, have impacted Gross Margin]

🛍️ Overheads

  • [Overview of Overheads, including total amount, change from previous month, and percentage change]
  • [Highlight the one most Significant change in Overhead subcategories, with specific amount and percentage change]

🏦 Net Profit and Retained Earnings

  • [Key point about net profit and retained earnings]
  • [Explanation of how changes in Revenue, direct costs and Overheads, including significant transactions, have impacted Net Profit and Retained Earnings]
Ensure that the Executive Summary provides a high-level overview of the most important points from your detailed analysis, including the impact of any significant or anomalous transactions. ``` Key aspects of the prompt include: 1. **Context Provision**: The prompt includes processed financial data, account group details, and cost change analysis. 2. **Specific Instructions**: It provides clear guidelines on what to focus on in the analysis, such as revenue, cost of delivery, and significant transactions. 3. **Output Structuring**: The prompt specifies the desired HTML structure for the output, ensuring a consistent and well-organized report. 4. **Analytical Guidance**: It provides guidance on how to approach the analysis, including comparing current performance to previous periods and identifying significant changes. 5. **Customized Formatting**: Instructions for using specific styles and tags in the output are included. The notebook example also stores the calculation table output and the commentary in a BigQuery table so that we can then include it in our Looker financial dashboard to run alongside our actual monthly P&L report, using a Looker custom visualization we’ve developed that allows us to display the formatted HTML that our LLM produces. In our internal version of this notebook we’ve also extended the financial analysis performed by the notebook and LLM to include highlighting new recurring payments, forecasting our cashflow for the coming months and other key parts of our monthly financial reporting process. For more examples of what we’re currently doing with GenAI within Rittman Analytics check out our other blogs below: - [Is Your Business Ready and Enabled For Generative AI?](https://www.rittmananalytics.com/blog/2024/8/2/is-your-business-ready-and-enabled-for-generative-ai) - [Creating a Conversational Data Analyst Chatbot using LangChain, OpenAI, Google BigQuery and Google Cloud Functions](https://www.rittmananalytics.com/blog/2024/7/5/ai-powered-conversational-data-analyst-chatbot) - [Automating Insights and Dashboard Report-Level Narratives using Vertex AI Gemini 1.5-Flash, Looker Studio and Google BigQuery BQML](https://www.rittmananalytics.com/blog/2024/6/17/automated-insights-gemini-1-5-flash) - [Automate your Contacts List Segmentation using Google BigQuery, Vertex AI and the ML.GENERATE\_TEXT function](https://www.rittmananalytics.com/blog/2024/1/14/automate-your-contacts-list-segmentation-using-google-bigquery-vertex-ai-and-the-mlgeneratetext-function) - [Generative AI Comes to Looker via Vertex AI and BigQuery BQML](https://www.rittmananalytics.com/blog/2024/2/5/generative-ai-comes-to-looker-via-vertex-ai-and-bigquery-bqml) or take a look at our [Generative AI services page](https://www.rittmananalytics.com/services/genai) and our fixed-price, [Jumpstart GenAI and Looker consulting package](https://www.rittmananalytics.com/packages/p/looker-genai-jumpstart-package-7sk3b) that can deliver these types of AI-powered dashboards for your organization, now! ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2Ffc8d3-0b27183a-8ff2-4221-845b-843841d30614_1000x620.png%3Fssl%3D1) # **INTERESTED? FIND OUT MORE!** Rittman Analytics is a boutique data analytics consultancy that helps ambitious, digital-native businesses scale-up their approach to data, analytics and [generative AI](https://www.rittmananalytics.com/services/genai). We’re authorised delivery partners for [Google Cloud](https://www.rittmananalytics.com/partners/looker) along with [Oracle](https://www.rittmananalytics.com/partners/oracle), [Segment](https://www.rittmananalytics.com/partners/segment), [Cube](https://www.rittmananalytics.com/partners/cube), [Dagster](https://www.rittmananalytics.com/partners/dagster), [Preset](https://www.rittmananalytics.com/partners/preset), [dbt Labs](https://www.rittmananalytics.com/partners/dbt-labs) and [Fivetran](https://www.rittmananalytics.com/partners/fivetran) and are experts at helping you design an analytics solution that’s right for your organisation’s needs, use-cases and budget and working with you and your data team to successfully implement it. If you’re looking for some help and assistance with your AI initiative or would just like to talk shop and share ideas and thoughts on what’s going on in your organisation and the wider data analytics world, [contact us now](https://calendly.com/markrittman/initial-discovery-call-with-mark-rittman?utm_source=medium.com&utm_medium=blog&utm_campaign=customer_segmentation) to organise a 100%-free, no-obligation call — we’d love to hear from you! --- # Roll with It with Cube, Embeddable and Rittman Analytics on Monday, 16th September 2024 : London RSA House, Durham Street Auditorium Canonical: https://rittmananalytics.com/blog/2024/09/05/roll-with-it-with-cube-embeddable-and-rittman-analytics-on-monday-16th-september-2024-london-rsa-house-durham-street-auditorium URL: https://rittmananalytics.com/blog/2024/09/05/roll-with-it-with-cube-embeddable-and-rittman-analytics-on-monday-16th-september-2024-london-rsa-house-durham-street-auditorium Description: So even bigger news than next year’s Oasis reunion concerts at Wembley: Cube.dev just recently announced Rollup, Cube’s first in-person user event to be held at RSA House Durham Street… Author: Mark Rittman Published: 2024-09-05T14:19:21+00:00 Modified: 2024-09-05T14:19:21+00:00 --- ![Roll with It with Cube, Embeddable and Rittman Analytics on Monday, 16th September 2024 : London RSA House, Durham Street Auditorium](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2Fe4db8-d4ba2d3e-7b5f-4826-bfec-7b3807ab5f91_1000x818.png%3Fssl%3D1) ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2Fe4db8-d4ba2d3e-7b5f-4826-bfec-7b3807ab5f91_1000x818.png%3Fssl%3D1) So even bigger news than next year’s Oasis reunion concerts at Wembley: [Cube.dev](https://cube.dev/) just recently announced [Rollup, Cube’s first in-person user event](https://cube.registration.goldcast.io/events/7d4eeddd-e87a-44bc-940a-25274a5f5a70) to be held at [RSA House Durham Street Auditorium](https://www.thersa.org/rsa-house/our-spaces/durham-street-auditorium) at 8 John Adam Street, London, WC2N 6EZ on Monday, 16th September 2024 – and we’ll be speaking at the event. Rollup is Cube’s first in-person user event, where you’ll be able to discover how a universal semantic layer can tackle current and future data problems and help data teams work better together. You’ll connect with industry leaders, mingle with other Cube enthusiasts and be the first to experience Cube’s latest innovations – and we’ll be speaking along with our mutual partners Embeddable about the work we’ve been doing with clients such as Osprey Charging and Facility Solutions Group, Inc. The event runs in the morning starting with breakfast at 10am, with an agenda as follows: - 10:30 – 10.35 : Welcome to Rollup: Bringing Cube Users Together (David Jayatillake, VP of AI, Cube) - 10:35 – 11:30 : Rollup Keynote: Build AI & Analytics Solutions with Unified Data (David Jayatillake and Igor Lukanin, Head of Product, Cube) - **11:30 – 11:55 : Best in Class BI: Elevate Your Business Insights with Rittman Analytics and Cube (Mark Rittman, Founder & CEO, Rittman Analytics)** - 11:55 – 12:20: Incredible Embeddable: Embedded Analytics Powered by Cube (Tom Gardiner, Founder & CEO, Embeddable) - 12:20 – 12:45: Customer Success Story: Permutive & Cube (Guy Dawson, Permutive) - 12:45 – 13:00: Final Words & Last Musings (David Jayatillake) Rollup is free to attend but attendee numbers are limited, so don’t look back in anger and leave registration too late – [event and registration details are here](https://cube.registration.goldcast.io/events/7d4eeddd-e87a-44bc-9). --- # Is Your Business Ready and Enabled For Generative AI? Canonical: https://rittmananalytics.com/blog/2024/09/02/is-your-business-ready-and-enabled-for-generative-ai URL: https://rittmananalytics.com/blog/2024/09/02/is-your-business-ready-and-enabled-for-generative-ai Description: Before you dive into your first generative AI (or “Gen AI”) implementation it’s crucial to have a clear understanding of how this technology will benefit your business, how it will… Author: Mark Rittman Published: 2024-09-02T21:49:00+00:00 Modified: 2024-09-02T21:49:00+00:00 --- ![Is Your Business Ready and Enabled For Generative AI?](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2Fe7d87-b16784bf-e6bc-4733-a76d-7a5bad8c6a56_1000x597.jpeg%3Fssl%3D1) ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2Fe7d87-b16784bf-e6bc-4733-a76d-7a5bad8c6a56_1000x597.jpeg%3Fssl%3D1) Before you dive into your first [generative AI](https://www.rittmananalytics.com/offers/artificial-intelligence) (or “Gen AI”) implementation it’s crucial to have a clear understanding of how this technology will benefit your business, how it will impact on your operations and whether you have the culture, skills and capability to deliver it effectively. Every business and organisation we speak to wants to adopt Gen AI — to drive business value, improve decision-making processes and to stay competitive when everyone else in their industry is adopting this game-changing new technology. For example we’ve just been helping a major UK retailer adopt Gen AI to [automate the delivery of actionable insights](https://www.rittmananalytics.com/blog/2024/6/17/automated-insights-gemini-1-5-flash) to their regional sales team, and we’ve just enabled a major SaaS entertainment platform to evaluate [conversational agents powered by Gen AI](https://www.rittmananalytics.com/blog/2024/7/5/ai-powered-conversational-data-analyst-chatbot) as way of providing data and analytics to promoters while reducing their operational costs. Take our [online quiz](https://go.rittmananalytics.com/genai) to see how well prepared you and your business are for adopting Gen AI, or read the rest of this article and find out what you need to know — and how we can help you get started and your first prototype deployed. So what does it take to make a success of your first Gen AI implementation? Here’s what we’ve learned makes the difference between project success or failure… ### **1\. Do You Have A Clear Set of Use-Cases and Organisational Goals?** Before diving headfirst into generative AI, it’s essential to have a clear roadmap. This means identifying specific, value-driven use cases that align with your business objectives. Ask yourself:  - Does your organization have specific, high-value applications for generative AI that align with your strategic goals? - To what extent have you involved key stakeholders from various departments in defining these use cases? - Have you conducted a thorough cost-benefit analysis for each proposed generative AI initiative? - What measures have you taken to ensure legal and ethical compliance in your planned AI implementations? - How do you plan to measure and evaluate the success of your generative AI projects? Without clearly defined use cases, organizations risk implementing generative AI haphazardly, leading to wasted resources, misaligned efforts, and failure to address critical business needs or capitalize on strategic opportunities ### **2\. Does your Business Have Sufficient Analytics Maturity and Capability?** A strong foundation in data analytics is essential for successful generative AI adoption. Ask yourself questions such as: - How would you rate the integration and accessibility of data across your organization?  - What advanced analytics techniques (e.g., predictive modeling, machine learning) are currently in use within your company? - How robust is your data governance framework, particularly concerning data quality and security? - To what extent does your team leverage cloud-based analytics platforms and tools? - How frequently do you update and refine your analytics processes and technologies? Lack of analytics maturity can severely limit an organization’s ability to effectively leverage generative AI, potentially resulting in inaccurate insights, data quality issues, and an inability to scale AI initiatives across the enterprise ### 3\. Does Your Organization Have and Encourage a Culture of Innovation? A forward-thinking, innovative culture is vital for embracing generative AI. Consider these aspects of your organizational culture. Ask yourself: - How does your leadership team actively promote and support innovative initiatives? - What processes are in place to encourage and evaluate new ideas from employees at all levels?  - How do you balance the need for innovation with maintaining operational stability and efficiency? - To what extent are teams empowered to experiment with new technologies and approaches? - How do you recognize and reward innovative thinking and risk-taking within your organization? Without a culture that embraces innovation, organizations may face resistance to change, struggle with adoption of new technologies, and miss out on the transformative potential of generative AI, ultimately falling behind more agile competitors. ### **4\. Do You Have the Right Set of Skills Within Your Team?** The success of generative AI initiatives heavily depends on having a skilled and adaptable workforce. Assess your team’s capabilities by asking yourself: - How would you evaluate your team’s current expertise in AI and machine learning technologies? - What ongoing training or upskilling programs do you have in place for AI-related skills? - How proficient are your team members in communicating complex AI concepts to non-technical stakeholders? - To what extent is your team familiar with AI ethics and responsible AI practices? - Have you identified any skill gaps that need to be addressed to support your generative AI initiatives? Failing to cultivate the necessary skills within your team can lead to poor implementation, misinterpretation of AI-generated insights, and ethical missteps, potentially undermining the credibility and effectiveness of your generative AI initiatives ### INTERESTED? FIND OUT MORE! Rittman Analytics is a boutique data analytics consultancy that helps ambitious, digital-native businesses scale-up their approach to data, analytics and [generative AI](https://www.rittmananalytics.com/services/genai). We’re authorised delivery partners for [Google Cloud](https://www.rittmananalytics.com/partners/looker) along with [Oracle](https://www.rittmananalytics.com/partners/oracle), [Segment](https://www.rittmananalytics.com/partners/segment), [Cube](https://www.rittmananalytics.com/partners/cube), [Dagster](https://www.rittmananalytics.com/partners/dagster), [Preset](https://www.rittmananalytics.com/partners/preset), [dbt Labs](https://www.rittmananalytics.com/partners/dbt-labs) and [Fivetran](https://www.rittmananalytics.com/partners/fivetran) and our **Data to AI Quickstart** package can help you prepare and plan, enable your team and implement your first generative AI solution. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2Fe07d6-2fe11b41-bf63-4d2b-afb2-652e794e896c_1000x618.png%3Fssl%3D1) Over four weeks we’ll work with your business to: - **Inform** your team on the latest in Generative AI for analytics - Work with stakeholders to define your **Gen AI strategy** - Identify **high-impact use cases** specific to your business - Prototype an **innovative proof-of-concept** to validate the use-case - **Enable your team** by co-developing and implementing the solution Check-out our [Generative AI Services page](https://www.rittmananalytics.com/offers/artificial-intelligence) on our website to discover our Gen AI use-cases or [contact us now](https://calendly.com/markrittman/initial-discovery-call-with-mark-rittman?utm_source=medium.com&utm_medium=blog&utm_campaign=customer_segmentation) to organise a 100%-free, no-obligation discovery call to discuss your needs in more detail — we’d love to hear from you! --- # Announcing our Automated dbt-to-Dataform Migration Tool, Powered by Open AI Canonical: https://rittmananalytics.com/blog/2024/08/01/announcing-our-automated-dbt-to-dataform-migration-tool-powered-by-opennbspai URL: https://rittmananalytics.com/blog/2024/08/01/announcing-our-automated-dbt-to-dataform-migration-tool-powered-by-opennbspai Description: At Rittman Analytics we’ve been advocates of dbt (“Data Build Tool”) and a partner of dbt Labs since back when they were known as Fishtown Analytics and dbt was their… Author: Mark Rittman Published: 2024-08-01T13:02:55+00:00 Modified: 2024-08-01T13:02:55+00:00 --- ![Announcing our Automated dbt-to-Dataform Migration Tool, Powered by Open AI](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2F9c978-a45b75eb-16f5-4781-ae44-e554f10f8007_1000x402.png%3Fssl%3D1) At Rittman Analytics we’ve been [advocates of dbt](https://www.rittmananalytics.com/partners/dbt-labs) (“Data Build Tool”) and [a partner of dbt Labs](https://blog.rittmananalytics.com/rittman-analytics-is-now-a-uk-consulting-partner-for-dbt-data-build-tool-2f23ac507d15) since back when they were known as Fishtown Analytics and dbt was their project accelerator toolkit. Increasingly though and especially for clients who are invested in the Google Cloud ecosystem we’re being asked to use [Dataform](https://cloud.google.com/dataform?hl=en), rather than dbt, to perform these tasks.  ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2F9c978-a45b75eb-16f5-4781-ae44-e554f10f8007_1000x402.png%3Fssl%3D1) Dataform is a SQL-based data transformation tool, similar to dbt, but intended these days to work natively with Google BigQuery and run as a fully-managed service within Google Cloud.  Crucially though for Google Cloud customers whilst Dataform Core, like dbt Core, is [free to use](https://cloud.google.com/dataform/pricing) any supporting services spend on scheduling and orchestration, for example, comes off of their existing committed Google Cloud spend rather than having to pay-for (and agree legals for) a separate service such as dbt Cloud. A comparison of Dataform to dbt is something for another blog post but for organizations such as ours with considerable investment and [code libraries](https://github.com/rittmananalytics/ra_attribution) written for dbt, having to convert those by-hand to Dataform is quite a daunting task. For example; - while both dbt and Dataform have the concept of models which in Dataform are called “tables”, Dataform’s syntax is of course different and tests, for example, are defined in-line rather than in separate .yml properties files - dbt models are defined using SQL and Jinja code, whereas Dataform tables are defined using SQL and Javascript code - dbt project configurations are stored in dbt\_project.yml and defined in YAML whereas Dataform configurations go into either dataform.json in JSON format, or more recently workflow\_settings.yaml stored in YAML format; and so on. To help us translate our dbt repositories as well as public ones such as [Jaffle Shop](https://github.com/dbt-labs/jaffle-shop) into Dataform syntax and configuration settings, we’ve been building a [dbt to Dataform Migration Tool](https://github.com/rittmananalytics/ra_dbt_to_dataform) that partially automates the process of converting dbt (data build tool) projects to Dataform, focusing on BigQuery as the data warehouse and available as [open-source Python code](https://github.com/rittmananalytics/ra_dbt_to_dataform) on our Github site. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2F895c0-9b367050-697d-40f3-9aa3-fbf4d67f3067_1000x689.png%3Fssl%3D1) It’s not yet a turn-key solution that handles all aspects of the migration for you, but it will take care of the routine steps so that you can concentrate on manually converting the more complex parts; in its initial 0.1.0-alpha release it: - Converts dbt models to Dataform SQLX files, with limitations - Translates dbt source definitions to Dataform declarations - Converts dbt macros to Dataform functions using GPT-4 (requires OpenAI API key) - Preserves project structure, adapting it to Dataform best practices - Handles (with limitations) dbt-specific Jinja syntax and converts it to JavaScript - Supports conversion of dbt variables to Dataform project config variables - Automatically converts common dbt\_utils functions to their BigQuery equivalents - Uses GPT-4 to check and correct Dataform syntax in converted files (requires OpenAI API key) - Generates a detailed conversion report highlighting potential issues and syntax corrections So how does this migration tool work, and how does it use the OpenAI API to assist in some of the more complex migration tasks? Our approach to automating this code conversion was to use a combination of rule-based transformations and AI-powered processing for more complex scenarios. This hybrid approach enables the tool to handle both straightforward translations and nuanced, context-dependent conversions. While the process is largely automated, it is designed to complement rather than replace human expertise. The tool provides a solid foundation for migration, but user intervention will be necessary for project-specific issues and handling of unsupported features. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2F75f45-32032489-9c11-4532-947d-4602f38e570c_1000x159.png%3Fssl%3D1) The migration process is made-up of seven steps, plus manual review and completion by yourself afterwards. 1. The **RepositoryAnalyzer** modules scans your dbt project structure to identify models, tests, macros, and YAML files. 2. The **ProjectConfigConverter** modules translates your dbt\_project.yml package settings to the equivalent dataform.json format, configuring your target warehouse connection settings and declaring project-wide settings and variables. 3. The **SourceConverter** module processes dbt source definitions and creates individual .sqlx source table definitions for each discovered source. 4. The **ModelConverter** module then translates each dbt SQL model to a Dataform .sqlx file, automatically handling reference conversions, variable replacements, and macro translations. 5. The **MacroConverter** module then transforms dbt macros into Dataform JavaScript functions by sending them, wrapped in a prompt, to the Open AI API for conversion, like this: ``` Convert the following dbt macro to a JavaScript function for Dataform: {dbt_macro_content} Follow these guidelines: 1. Convert Python/Jinja syntax to JavaScript. 2. Replace dbt-specific functions with Dataform equivalents where possible. 3. For SQL generation, use JavaScript template literals. 4. If there's no direct Dataform equivalent for a dbt function, implement the functionality in JavaScript. 5. Add any necessary comments or explanations. Provide only the converted JavaScript function: ``` 6\. Model and metadata conversions are then passed to the **SyntaxChecker** module which again uses the OpenAI API, this time to review each converted model against Dataform syntax rules and best practices, like this: ``` Check if the following Dataform SQLX code is valid. If it's not valid, correct it and explain the changes made. If it's valid, just respond with "Valid". Always include the full corrected code in your response, even if only small changes were made. Wrap the corrected code in ```sqlx and ``` tags. Code: {sqlx_content} ``` 7\. Finally, the **ConversionReport** module outputs a report on the migration process, highlighting potential issues, syntax corrections, and areas needing manual review. Current limitations in the migration process are: - Complex dbt macros may require manual adjustment after conversion - Custom dbt tests might need additional implementation in Dataform - The tool assumes a BigQuery setup; adjustments may be needed for other warehouses - Certain dbt-specific features, such as dbt’s semantic layer, might not have direct equivalents in Dataform and are therefore out-of-scope - While this converter handles many aspects of dbt projects, some features are not currently supported or require manual intervention including seeds, snapshots, custom tests, documentation, exposures and advanced hooks, though we do intent to cover most of these off over time The python package and all its code are available as open-source at [https://github.com/rittmananalytics/ra\_dbt\_to\_dataform](https://github.com/rittmananalytics/ra_dbt_to_dataform) and we’d welcome pull requests and issue reports.  And … like London buses where there’s none for ages and then two turn-up at once, my old friend Christian Screen [released his own dbt to Dataform converter](https://www.linkedin.com/posts/christianscreen_introducing-the-dbt-to-dataform-converter-activity-7221182790409584640-22Ir/) a week or so ago, so check his out as well and I’m sure there’ll be some cross-project collaboration at some point. ### INTERESTED? FIND OUT MORE! Rittman Analytics is a boutique data analytics consultancy that helps ambitious, digital-native businesses scale-up their approach to data, analytics and [generative AI](https://www.rittmananalytics.com/services/genai). We’re authorised delivery partners for [Google Cloud](https://www.rittmananalytics.com/partners/looker) along with [Oracle](https://www.rittmananalytics.com/partners/oracle), [Segment](https://www.rittmananalytics.com/partners/segment), [Cube](https://www.rittmananalytics.com/partners/cube), [Dagster](https://www.rittmananalytics.com/partners/dagster), [Preset](https://www.rittmananalytics.com/partners/preset), [dbt Labs](https://www.rittmananalytics.com/partners/dbt-labs) and [Fivetran](https://www.rittmananalytics.com/partners/fivetran) and are experts at helping you design an analytics solution that’s right for your organisation’s needs, use-cases and budget and working with you and your data team to successfully implement it. If you’re looking for some help and assistance with your Dataform initiative or would just like to talk shop and share ideas and thoughts on what’s going on in your organisation and the wider data analytics world, [contact us now](https://calendly.com/markrittman/initial-discovery-call-with-mark-rittman?utm_source=medium.com&utm_medium=blog&utm_campaign=customer_segmentation) to organise a 100%-free, no-obligation call — we’d love to hear from you! --- # Making Smart Buildings Smarter for Facility Solutions Group in Partnership with Embeddable Canonical: https://rittmananalytics.com/blog/2024/07/10/making-smart-buildings-smarter-for-facility-solutions-group-with-embeddable URL: https://rittmananalytics.com/blog/2024/07/10/making-smart-buildings-smarter-for-facility-solutions-group-with-embeddable Description: At Rittman Analytics, we’re always excited to partner with innovative companies that are pushing the boundaries of what’s possible with data and analytics. That’s why we’re thrilled to spotlight our… Author: Mark Rittman Published: 2024-07-10T13:04:16+00:00 Modified: 2024-07-10T13:04:16+00:00 --- ![Making Smart Buildings Smarter for Facility Solutions Group in Partnership with Embeddable](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2F3a69e-89511779-9123-4493-ba99-174633ac304e_1000x701.png%3Fssl%3D1) ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2F3a69e-89511779-9123-4493-ba99-174633ac304e_1000x701.png%3Fssl%3D1) At Rittman Analytics, we’re always excited to partner with innovative companies that are pushing the boundaries of what’s possible with data and analytics. That’s why we’re thrilled to spotlight our [collaboration with Embeddable](https://www.rittmananalytics.com/partners/embeddable), a powerful embedded analytics platform that’s becoming increasingly popular with organisations looking to integrate data visualizations and reporting capabilities into their SaaS applications. #### Embeddable: The Developer’s Toolkit for Seamlessly-Integrated Analytics Experiences Embeddable offers a unique approach to embedded analytics, providing developers with the tools to create beautiful, pixel-perfect dashboards that seamlessly blend with their existing applications. By leveraging Embeddable’s technology, teams can build fast, interactive, and fully-custom analytics experiences in just a fraction of the time it would typically take. Key features of Embeddable include: 1. No-code Dashboard Builder: Easily create and customize dashboards without the need for coding or IT support. 2. Native Look and Feel: Data visualization components that can be defined and edited in code to match your application’s exact design. 3. Scalability and Integration: Connects to popular databases and cloud data warehouses with an embedded business semantic layer. 4. Cube Integration: Utilizes Cube’s open-source semantic model and caching service for optimal performance and data consistency. #### Case Study: FSG Smart Buildings Transforms Facility Management We recently partnered with Embeddable [to enable Austin, Texas-based Facility Solutions Group (FSG)](https://www.rittmananalytics.com/case-studies/fsg) to surface key performance indicators (KPIs) and reporting tables in Chariot, their client-facing Building Automation System. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2F28177-49755856-a3c7-47bb-b0e8-d24211fabbfd_1000x1212.png%3Fssl%3D1) FSG needed to make their collected data more actionable and easier to consume for their clients, particularly those in the convenience store sector. Their existing system required clients to have spreadsheet skills and work with fragmented data extracts, limiting their ability to gain a holistic view of business operations, tenant behavior, and financial performance. Rittman Analytics was engaged to lead the technical implementation of Embeddable and guide FSG’s team through the process of delivering embedded application analytics. Our approach included: 1. Creating placeholder Embeddable KPIs that could be surfaced in advance of finalizing specific business logic. 2. Embedding these KPIs in a prototype version of the customer-facing application. 3. Collaborating with Embeddable to define requirements and a development roadmap. 4. Modeling and warehousing FSG’s high-volume smart buildings event data. 5. Defining a rich, descriptive business semantic layer over the warehoused data. 6. Orchestrating and scheduling the data pipeline using Dagster Cloud for up-to-date client data. The implementation of Embeddable, guided by Rittman Analytics, significantly improved how FSG’s clients interact with their data. The solution provided a holistic view of business operations, tenant behavior, and financial performance, enhancing operational efficiency and enabling more effective buildings management. Adam Haynes, Chief Product Officer at FSG, praised the collaboration: *“Rittman Analytics was able to do what seemed impossible. With only hours of spin-up, they dove head-first into the work with our team at FSG. Their incredible team brought structure and strategy to the newly developing discipline of data analytics within FSG.”* ### INTERESTED? FIND-OUT MORE! As [partners with Embeddable](https://www.rittmananalytics.com/partners/embeddable) and [Cube](https://www.rittmananalytics.com/partners/cube), Rittman Analytics offers specialized services to help your team prepare your data for use with Embeddable visualizations. Our [Embeddable Quickstart Packages](https://www.rittmananalytics.com/packages/embeddable-jump-starts) are designed to get you up and running quickly, with options tailored to your specific needs and budget. Whether you’re looking to protect your app response times, structure your data for analytics use, report across multiple sources, ensure dashboard accuracy, or scale your application dashboards, our team has the expertise to guide you through the process. Ready to transform your data into actionable insights with Embeddable? [Contact us today](https://calendly.com/markrittman/initial-discovery-call-with-mark-rittman) to learn more about our Embeddable Quickstart Packages and how we can help you build remarkable analytics experiences for your business. --- # Creating a Conversational Data Analyst Chatbot using LangChain, OpenAI, Google BigQuery and Google Cloud Functions Canonical: https://rittmananalytics.com/blog/2024/07/05/ai-powered-conversational-data-analyst-chatbot URL: https://rittmananalytics.com/blog/2024/07/05/ai-powered-conversational-data-analyst-chatbot Description: In our previous post on Automating Insights and Dashboard Report-Level Narratives using Vertex AI Gemini 1.5-Flash, Looker Studio and Google BigQuery BQML I showed how generative AI could be used to… Author: Mark Rittman Published: 2024-07-05T22:52:34+00:00 Modified: 2024-07-05T22:52:34+00:00 --- ![Creating a Conversational Data Analyst Chatbot using LangChain, OpenAI, Google BigQuery and Google Cloud Functions](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2Fab115-ebf500fa-d438-4a1a-9095-163c8ea8408d_1000x404.png%3Fssl%3D1) In our previous post on [Automating Insights and Dashboard Report-Level Narratives using Vertex AI Gemini 1.5-Flash, Looker Studio and Google BigQuery BQML](https://www.rittmananalytics.com/blog/2024/6/17/automated-insights-gemini-1-5-flash) I showed how generative AI could be used to add commentary and action points to individual charts and tables in a Looker Studio report. In this post I’ll show how we’ve combined the thinking and reasoning part of generative AI with conversational, chatbot-style interface to create a “proof of concept” AI-powered data analyst website plugin, able to answer questions about KPIs and trends within your data warehouse and with the ability to understand the context and meaning in the questions your users ask of it. To create this conversational data analyst chatbot we created a simple Javascript client front-end that collects users’ questions and sends them, via a REST API interface to a Google Cloud Function that, under-the-covers, uses LangChain and OpenAI’s GPT-4-Turbo LLM to query a Google BigQuery cloud data warehouse. When clicked, the chatbot starts by asking if you’d like to ask a question about your data, for example by asking what was our total revenue in May 2024 and pressing the “go” button. After first responding with “thinking…”, around twenty seconds later the chatbot comes back with an answer, to which you can either respond with further questions or press the close (“x”) button to end the chatbot session. To make those questions and answers conversational, the Javascript front-end remembers the answers given the user within each chat session and then adds them to the user’s question when sending them to the LLM for processing, so that if you follow-up question asks “how does this compare to same time last year” it understands that “this” is a reference to revenue and “same time last year” is a reference to May 2023. To answer these questions the chatbot sends SQL queries to our Google BigQuery data warehouse in which we’ve created a set of summary tables specifically designed to answer questions around monthly performance KPIs, sales and lead generation activity and data on sessions and page views for our company website. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2Fab115-ebf500fa-d438-4a1a-9095-163c8ea8408d_1000x404.png%3Fssl%3D1) So how did we put this together? If you’re interested in seeing the code and maybe running it yourself there’s a [Github repo](https://github.com/rittmananalytics/ra-databot) with instructions on how to deploy to Google Cloud, and of course if you’d like us to deploy or build something similar for your organization then check-out the section at the end of this post on how to contact us. At a high-level though there’s two main parts to this conversational data analyst chatbot: 1. The back-end, written in Python and running as a serverless [Google Cloud Function](https://cloud.google.com/functions?hl=en) in Google Cloud Platform, that receives questions from a front-end service and sends-back answers for that front-end to process and display 2. The front-end, which can be as simple as a cURL command such as the one below: ``` curl -X POST -H "Content-Type: application/json" \ --data '{"question":"and what was it in April 2024?"}' \ https://YOUR_REGION-YOUR_PROJECT_ID.cloudfunctions.net/ra-databot ``` or as shown in the diagram below, could be a Javascript + CSS web page chatbot dialog that shows those questions and answers in a conversational, messaging app format. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2Ff50ca-9e7694b0-445f-4ee4-995d-456a77d11a1f_1000x850.png%3Fssl%3D1) So how does the front-end client and back-end service work? 1\. The chatbot dialog is displayed when the user presses a chatbot button that triggers the display of the chatbot dialog. 2\. The user types in a question into the chatbot’s “type question here” text box, for example “How much were our sales in May 2024?” 3\. Javascript within that chatbot plugin then sends the question, together with any previous answers to that user’s question in this particular chat session, to the back-end service’s REST API like this: ``` fetch('https://YOUR_REGION-YOUR_PROJECT_ID.cloudfunctions.net/ra-databot', requestOptions) .then((response) => response.text()) .then((data) => { console.log('Received data:', data); tempBubble.innerHTML = data; // Replace the temporary bubble content with the actual response chatHistory += `Bot: ${data}\n`; // Append bot response to chat history }) .catch((error) => { console.error('Error:', error); tempBubble.innerHTML = 'Sorry, I could not get the answer. Please try again later.'; }); } }); ``` 4\. The back-end service takes this incoming question and prepends the following prompt to the user’s question: ``` “You are a knowledgeable data analyst working for Rittman Analytics. Answer questions correctly, do not delete or alter any data and provide concise (no more than 10 words) commentary and analysis where appropriate. Use the ra-development.analytics_wide.monthly_company_metrics for monthly summary KPI questions, ra-development.analytics_wide.sales_leads for questions about sales leads, ra-development.analytics_wide.website_traffic for questions about website performance, ra-development.analytics_wide.sales_deals for sales pipeline and sales activity questions to answer this question, and no other tables. Do not include markdown-style triple backticks in the SQL you generate and try to use or validate. Question is:” ``` 5\. This prompt prepended to the user’s question is then passed to [LangChain](https://www.langchain.com/), a framework designed to simplify the integration of large language models (LLMs) like OpenAI’s [GPT-4-Turbo](https://help.openai.com/en/articles/8555510-gpt-4-turbo-in-the-openai-api) with various data sources and tools, enabling the creation of sophisticated applications that leverage natural language understanding and generation capabilities. LangChain acts as a bridge between the language model and the data sources/tools and provides abstractions and utilities to connect the LLM with various tools, including databases, APIs, and other services, as we do in our back-end code: ``` from google.cloud import bigquery from sqlalchemy import * from sqlalchemy.engine import create_engine from sqlalchemy.schema import * import os from langchain.agents import create_sql_agent from langchain.agents.agent_toolkits import SQLDatabaseToolkit from langchain.sql_database import SQLDatabase from langchain_openai import ChatOpenAI from langchain.agents import AgentExecutor from config.config import * project = os.environ["GCP_PROJECT"] dataset = os.environ["BQ_DATASET"] gcp_credentials = os.environ["GCP_CREDENTIALS"] open_ai_model = os.environ["OPEN_AI_MODEL"] sqlalchemy_url = f'bigquery://{project}/{dataset}?credentials_base64={gcp_credentials}' db = SQLDatabase.from_uri(sqlalchemy_url) llm = ChatOpenAI( model=open_ai_model, temperature=0, max_tokens=None, timeout=None, max_retries=2 ) toolkit = SQLDatabaseToolkit(db=db, llm=llm) agent_executor = create_sql_agent( llm=llm, toolkit=toolkit, verbose=True, max_iterations=20, top_k=1000, agent_type="tool-calling" ) Instruction = "..." ``` 6\. LangChain’s [SQL Agent](https://python.langchain.com/v0.1/docs/use_cases/sql/agents/) is designed to translate natural language queries into SQL queries by leveraging the capabilities of, for example, OpenAI’s GPT-4-Turbo LLM in order to understand the user’s question and generate the appropriate SQL query. 7\. The Tools within LangChain include connectors to cloud data warehouse such as Google BigQuery, enabling the execution of SQL queries and retrieval of data. Those tool invocations first read the database data dictionary, then choose the most suitable table or tables to query, then sample those tables’ contents and then write, test and then execute the correct SQL query to return the answer to the user’s query 8\. The results from the cloud data warehouse are processed and formatted and the GPT-4-Turbo LLM can further refine the results, providing a natural-language explanation or summary if needed. 9\. Those results are then sent-back to the Javascript app plugin as the REST API response and then displayed in the chatbot UI as the answer to the users’ question. 10\. If the user asks further questions as follow-ups to their first question, the Chatbot javascript app appends the results of previous questions in this chat to those follow-up questions so that the LLM has the context of the conversation available to it when formulating its answer. In-practice developing and deploying a Cloud Function needs a means to store and provide as required the API keys and credentials needed to access the Open AI API and the BigQuery data warehouse and the [readme](https://github.com/rittmananalytics/ra-databot/blob/main/README.md) file within our Github repo explains how we use Google Cloud’s Secrets Manager to do this. TODO on the development list includes securing the back-end’s REST API interface so that it can only be accessed by a service account with the right IAM permissions granted, and to enable the front-end client to pass the user’s ID along with their question so that answers are limited to just the data they’re allowed to see. We also configure the Cloud Function to have at least one instance running at any one time so that users don’t have to wait around for python libraries to load-up if the function has to do a cold start; longer-term we’d look into running the back-end code as a service deployed via Cloud Run if that works-out faster and cheaper to run than as a Cloud Function. ### **INTERESTED? FIND OUT MORE!** Rittman Analytics is a boutique data analytics consultancy that helps ambitious, digital-native businesses scale-up their approach to data, analytics and [generative AI](https://www.rittmananalytics.com/services/genai).  We’re authorised delivery partners for [Google Cloud](https://www.rittmananalytics.com/partners/looker) along with [Oracle](https://www.rittmananalytics.com/partners/oracle), [Segment](https://www.rittmananalytics.com/partners/segment), [Cube](https://www.rittmananalytics.com/partners/cube), [Dagster](https://www.rittmananalytics.com/partners/dagster), [Preset](https://www.rittmananalytics.com/partners/preset), [dbt Labs](https://www.rittmananalytics.com/partners/dbt-labs) and [Fivetran](https://www.rittmananalytics.com/partners/fivetran) and are experts at helping you design an analytics solution that’s right for your organisation’s needs, use-cases and budget and working with you and your data team to successfully implement it. If you’re looking for some help and assistance with your AI initiative or would just like to talk shop and share ideas and thoughts on what’s going on in your organisation and the wider data analytics world, [contact us now](https://calendly.com/markrittman/initial-discovery-call-with-mark-rittman?utm_source=medium.com&utm_medium=blog&utm_campaign=customer_segmentation) to organise a 100%-free, no-obligation call — we’d love to hear from you! --- # Hell Freezes-Over as Oracle Exadata, Autonomous Database (and OCI) become Available on Google Cloud Platform Canonical: https://rittmananalytics.com/blog/2024/06/25/hell-freezes-over-as-oracle-exadata-autonomous-database-and-oci-become-available-on-google-cloud-platform URL: https://rittmananalytics.com/blog/2024/06/25/hell-freezes-over-as-oracle-exadata-autonomous-database-and-oci-become-available-on-google-cloud-platform Description: - **Duet AI in Looker**: Introduction of Duet AI as an AI assistant within Looker to streamline analytics workflows, backed by powerful language models and ensuring high-quality outputs through prompt engineering and responsible AI practices. - **Generative AI for Self-Service BI**: Emphasis on revitalizing self-service BI by using generative AI to lower technical barriers and enable both technical and business users to perform more complex analyses with less effort. - **Vertex AI and Looker Integration**: Enhancements in custom AI experiences for BI users by integrating Looker with Vertex AI, facilitating a tighter connection with Google Cloud services and enabling the development of custom data experiences and applications. - **AI-Powered Visualizations and Insights**: Advanced visualizations and insights generation using AI to make data more interactive and intuitive, including natural language processing capabilities for querying and report generation. One of the first, early manifestations of adding GenAI into Looker is a [Looker extension](https://github.com/looker-open-source/extension-gen-ai) that’s available now for self-install through [Github](https://github.com/looker-open-source/extension-gen-ai) and will be on the Looker Marketplace for point-and-click installation soon. The extension is installed into a separate LookML project in your Looker instance and then connects to your BigQuery database, via BQML, to one of the [Vertex AI large language models](https://cloud.google.com/vertex-ai/docs/generative-ai/learn-resources) (LLMs). Once installed, one of the uses for the GenAI extension is to auto-create Looker explore queries based on natural language queries such as “*how many sessions and users did our website receive over the past three months, broken-down by channel?*”, or “*what were our top 10 website pages, measured by page views, over the past six months?*”, as shown in the two screenshots below. The other use for the extension is to point it to one of your existing dashboards, again which can source its data from BigQuery and other data sources, and have it summarise the results and provide analysis and guidance, as its doing in the screenshot below that the dashboard from our [Customer First Order Segmentation using Looker and Google BigQuery](https://www.rittmananalytics.com/blog/2022/12/21/customer-first-order-segmentation-using-looker-and-google-bigquery) blog. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752871882645-uq0c4c.png%3Fresize%3D6374%252C4946%26ssl%3D1) Under the covers this GenAI Looker extension uses the same Vertex AI and BigQuery approach that we used in our [Automate your Contacts List Segmentation using Google BigQuery, Vertex AI and the ML.GENERATE\_TEXT function](https://www.rittmananalytics.com/blog/2024/1/14/automate-your-contacts-list-segmentation-using-google-bigquery-vertex-ai-and-the-mlgeneratetext-function) blog. This time around we have Looker, through the GenAI extension application, constructing a prompt to send to Vertex AI through [ML.GENERATE\_TEXT](https://cloud.google.com/bigquery/docs/generate-text) by combining your natural language query with either the explore metadata for the first use-case or the contents of the dashboard page for the second, returning either a LookML query and some analysis or just some analysis, respectively. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752871882555-g45d4c.png%3Fresize%3D3820%252C4388%26ssl%3D1) The screenshots below show two more examples of the GenAI extension being used to analyze and summarise the contents of our dashboards, the one on the left summarising our performance a few years ago and the right-hand one containing website traffic metrics. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752871883280-13tciv.png%3Fresize%3D8012%252C6542%26ssl%3D1) It’s a relatively simple example of adding generative AI into the Looker experience and you have to be careful, for example, to get it to cross-check its calculations when asking it analyze both time-series and aggregated versions of the same metric. And of course, like all datasets that you’re planning on using for AI and GenAI use-cases, the dashboard and its underlying warehouse dataset need to have well-labelled and unambigously-defined LookML models and table schemas so that your LLM can properly understand the meaning of the data you’re providing it — as they always do when delivered as one of our client, and internal, delivery projects. ## **Interested? Find Out More!** Rittman Analytics is a boutique data analytics consultancy that helps ambitious, digital-native businesses scale-up their approach to data, analytics and AI. We’re authorised delivery partners for [Google Cloud](https://www.rittmananalytics.com/partners/looker) along with [Oracle](https://www.rittmananalytics.com/partners/oracle), [Segment](https://www.rittmananalytics.com/partners/segment), [Cube](https://www.rittmananalytics.com/partners/cube), [Dagster](https://www.rittmananalytics.com/partners/dagster), [Preset](https://www.rittmananalytics.com/partners/preset), [dbt Labs](https://www.rittmananalytics.com/partners/dbt-labs) and [Fivetran](https://www.rittmananalytics.com/partners/fivetran) and are experts at helping you design an analytics solution that’s right for your organisation’s needs, use-cases and budget and working with you and your data team to successfully implement it. If you’re looking for some help and assistance with your AI initiative or would just like to talk shop and share ideas and thoughts on what’s going on in your organisation and the wider data analytics world, [contact us now](https://calendly.com/markrittman/initial-discovery-call-with-mark-rittman?utm_source=medium.com&utm_medium=blog&utm_campaign=customer_segmentation) to organise a 100%-free, no-obligation call — we’d love to hear from you! [Looker](https://rittmananalytics.com/blog/tag/Looker)[Google Cloud Platform](https://rittmananalytics.com/blog/tag/Google+Cloud+Platform)[AI](https://rittmananalytics.com/blog/tag/AI)[Generative AI](https://rittmananalytics.com/blog/tag/Generative+AI) [Mark Rittman](https://rittmananalytics.com/blog?author=5846da08f5e231d83261e221) CEO of Rittman Analytics, host of the Drill to Detail Podcast, ex-product manager and twice company founder. [https://rittmananalytics.com](https://rittmananalytics.com/) --- # Automate your Contacts List Segmentation using Google BigQuery, Vertex AI and the ML.GENERATE_TEXT function — Rittman Analytics Canonical: https://rittmananalytics.com/blog/2024/01/14/automate-your-contacts-list-segmentation-using-google-bigquery-vertex-ai-and-the-ml-generate_text-function-rittman-analytics URL: https://rittmananalytics.com/blog/2024/01/14/automate-your-contacts-list-segmentation-using-google-bigquery-vertex-ai-and-the-ml-generate_text-function-rittman-analytics Description: Automate your Contacts List Segmentation using Google BigQuery, Vertex AI and the ML.GENERATE_TEXT function Artificial Intelligence 14 Jan Written By Mark Rittman When you’re trying to segment your contact list… Author: Mark Rittman Published: 2024-01-14T01:00:00+01:00 Modified: 2024-01-14T01:00:00+01:00 --- ![Automate your Contacts List Segmentation using Google BigQuery, Vertex AI and the ML.GENERATE_TEXT function — Rittman Analytics](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2024%2F01%2F1752871916208-4008ee.png%3Ffit%3D2500%252C2136%26ssl%3D1) # Automate your Contacts List Segmentation using Google BigQuery, Vertex AI and the ML.GENERATE\_TEXT function [Artificial Intelligence](https://rittmananalytics.com/blog/category/Artificial+Intelligence) 14 Jan Written By [Mark Rittman](https://rittmananalytics.com/blog?author=5846da08f5e231d83261e221) When you’re trying to segment your contact list in a CRM tool such as the one we use, HubSpot, one of the challenges is that only some of the data we hold on those contacts is by default available to us in their Hubspot profile. HubSpot, like most other CRM tools, captures and stores the basic details about a marketing contact such as their name, associated company and the campaign or website activity that introduced them to us, and enriches those basic details with public data such as their LinkedIn bio and Twitter handle. In previous blogs such as this one on [Customer 360](https://www.rittmananalytics.com/blog/2021/2/15/customer-360-degree-analysis-and-hightouch) and [How Rittman Analytics does Analytics Part 2 : Building our Modern Data Stack using dbt, Google BigQuery, Looker, Segment and Rudderstack](https://www.rittmananalytics.com/blog/2022/5/30/how-rittman-analytics-does-analytics-part-2-building-our-modern-data-stack-using-dbt-google-bigquery-looker-segment-and-rudderstack) I’ve talked about how we extract and centralise the data from our SaaS applications and centralize it in Google BigQuery, giving us a central source of contact data including: - Activity on Jira tickets by our own and client staff - Signing of statements of work by client sponsors - NPS survey scores and feedback given by client stakeholders - Summarised communications during the sales and delivery process - Ideal Customer Profile (ICP) scores and segments for each client ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752871916208-4008ee.png%3Fresize%3D5103%252C4359%26ssl%3D1) Combining this CRM and activity data for contacts together would, in-theory at least, allow us to add some useful and actionable segments to our contacts list including: - The types of analytics consulting services they’re likely to be interested in, based on information discovered in the sales process and projects delivered for that contact in the past - The size and nature of the organization they work for, allowing us define audiences of just contacts working for startups, for example - Whether the contact is likely to be the decision-maker or budget-holder for their organization, useful of course when working out who is the person who’s going to be signing-off your Statement of Work I said in-theory though as, whilst the cloud data warehouse makes this rich set of contact available available to marketers, actually deciding which segments to assign to each contact based on the data now available for each of them is quite a laborious, subjective and until now, manual process … in other words, one that’s ideal for automating using generative AI. The [ML.GENERATE\_TEXT](https://cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-generate-text) function that was recently added to Google BigQuery brings [ChatGPT-style generative AI](https://www.rittmananalytics.com/blog/2023/3/26/chatgpt-large-language-models-and-the-future-of-dbt-and-analytics-consulting) to Google SQL, making it possible to use natural language prompts to perform tasks in your data warehouse such as: - Classification - Sentiment Analysis - Entity extraction - Extractive Question Answering - Summarization - Rewriting text in a different style - Ad copy generation - Concept ideation Technically a BigQuery remote function call that, under the covers, leverages the various Google Cloud’s Vertex AI large language models (LLMs) to respond to your prompts, ML.GENERATE\_TEXT is one of the first manifestations of generative AI in Google BigQuery and one we’ll now use to automate this segmentation of our contacts list. If you’re familiar with ChatGPT and the way you use “prompts” to pose questions and get answers and explanations either as verbose text or more usefully for us, categorisations or summaries of your data, we need to take the same approach with ML.GENERATE\_TEXT to return the set of segment values for each contact. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752871916199-rho31.png%3Fresize%3D5500%252C3328%26ssl%3D1) Before we can start prompting our LLM and getting category predictions we first need to setup the connection to a Vertex AI LLM and make it available to BigQuery as a remote function endpoint. To do this, we: 1. Register the LLM we’re going to use in our Google Cloud project, [detailed steps for which](https://cloud.google.com/bigquery/docs/generate-text-tutorial#create_a_connection) are in the BigQuery docs and in-summary require you to: 2. Create a cloud resource link from BigQuery, recording the details of the service account and then grant the required IAM roles to this service account, as described in [this tutorial](https://cloud.google.com/bigquery/docs/generate-text-tutorial) 3. Register the Vertex AI text-bison remote model in your GCP project, using the connection name you created in the previous step and selecting `text-bison` as the model endpoint, as per [these docs](https://cloud.google.com/bigquery/docs/generate-text-tutorial) CREATE OR REPLACE MODEL \`ra-development.analytics\_ai.bison\_model\` REMOTE WITH CONNECTION \`ra-development.eu.ra-cloudai\` OPTIONS (ENDPOINT = 'text-bison'); 4\. Now we create an SQL query to join together all of the items of data available for each contact and concatenate those details to create a verbose “biography” for that contact that we’ll add to the end of the prompt when sending it to the LLM remote function; for example: WITH contact\_conversations AS ( SELECT conversation\_from\_contact\_name AS contact\_name, STRING\_AGG(conversation\_subject) AS contact\_conversation\_subjects FROM \`ra-development.analytics.conversations\_fact\` s GROUP BY 1 ), contact\_profile AS ( SELECT c.contact\_pk, c.contact\_name, -- Additional fields and complex COALESCE statements -- Truncated for brevity FROM \`ra-development.analytics.contacts\_dim\` c LEFT JOIN \`ra-development.analytics.contact\_companies\_fact\` cc ON c.contact\_pk = cc.contact\_fk AND contact\_next\_created\_date IS NULL LEFT JOIN contact\_conversations s ON c.contact\_name = s.contact\_name LEFT JOIN \`ra-development.analytics.companies\_dim\` d ON cc.company\_fk = d.company\_pk LEFT JOIN \`ra-development.analytics.contracts\_fact\` f ON c.contact\_pk = f.contact\_fk ), contact\_profiles\_deduped AS ( SELECT contact\_pk, contact\_name, company\_lifecycle\_stage, contact\_brief, company\_name, ideal\_customer\_group FROM contact\_profile GROUP BY 1,2,3,4,5,6 ), An example contact biography created by this SQL query is shown below, highlighting the elements of that particular biography that the LLM response is likely to consider in its categorisations: ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752871916323-f5tskf.png%3Fresize%3D3584%252C1760%26ssl%3D1) 5\. Next we’ll [design a text prompt](https://cloud.google.com/vertex-ai/docs/generative-ai/text/text-prompts) that will ask the LLM to [classify](https://cloud.google.com/vertex-ai/docs/generative-ai/text/text-prompts#classification_prompts) our contact by the person’s likely service interest, company size and likelihood of being a budget-holding decision maker, such as this one: > Classify this person as one of the following categories of probability of being a budget-holding decision maker in sales activity with us on behalf of their organisation,putting emphasis on their job title and whether they have signed statement of work or project delivery contracts before, (not employment, redundancy or related contracts) and with having no explicit job title reducing that probability, as either > – High > – Medium > – Low > > Also predict their categories of consulting services interest from the list > – “Operational Reporting” > – “Marketing Analytics” > – “Data Centralization” > – “Product Analytics” > – “Marketing Attribution” > – “Unknown” > > separating multiple categories if predicted for a contact using commas, and then predict the category of organization or company that the person works for from > – Startup, > – SMB, > – Mid-Market, > – Enterprise. > > Give me the rationale your categorisations ensuring any PII is obfuscated in that rationale and do not return any words or comments apart from these three sets of categorisations and rationale. Your response should be in the form of a single line of text that separates each categorisation for the person using a colon, for example: > High:Marketing Analytics,Product Analytics:SMB:Signed statement of work contracts before and has contacted about operational reporting related topics. > > Text:’ 3\. Then we decide on the [parameter settings](https://cloud.google.com/bigquery/docs/generate-text#generate_text) that will determine the type, length and predictability of the LLM’s response to our prompt, the most significant of which are listed below: - `max_output_tokens`: An INT64 parameter ranging from 1 to 1024, determining the model’s output length. Lower numbers yield brief responses, while higher settings allow for more extensive replies. By default it’s set at 50 and tokens, which are smaller than words and roughly equivalent to four characters, mean that 100 tokens translate to about 60-80 words. - `temperature`: A FLOAT64 parameter that can range between 0.0 and 1.0, influencing the randomness in the model’s responses especially when top\_k and top\_p are active. Lower temperatures suit straightforward, predictable replies while higher settings encourage varied and imaginative outputs. At 0, the model always chooses the most probable response, with 1.0 being the standard setting. - `top_k`: An INT64 parameter value adjustable from 1 to 40 with affects the model’s token selection randomness. Lower values lead to more predictable outputs whilst higher ones provide more varied responses, with the default being 40. A top\_k of 1 picks the most likely token while a setting of 3 chooses from the top three, incorporating the temperature setting for selection. For each token choice, the model samples from the top\_k most probable tokens, further refining them using the top\_p value and selecting the final token based on temperature. - `top_p`: Set as a FLOAT64 in the range of 0.0 to 1.0, this parameter also steers the randomness of the model’s responses. It’s set at 1.0 by default. The model selects tokens from the most to the least probable, as determined by top\_k, until their combined probability reaches the top\_p value. For example, with a top\_p of 0.5, if tokens A, B, and C have probabilities of 0.3, 0.2, and 0.1 respectively, the model will choose between A and B, using the temperature value, and exclude C. As we want the LLM request to return predictable replies, not random and imaginative ones and we want the rationale for the LLM’s predictions to be reasonably concise we choose 0.0 as the temperature and 250 as the max\_output\_tokens and add the call to ML.GENERATE\_TEXT to our SQL query, like this: WITH contact\_conversations AS ( SELECT conversation\_from\_contact\_name AS contact\_name, STRING\_AGG(conversation\_subject) AS contact\_conversation\_subjects FROM \`ra-development.analytics.conversations\_fact\` s GROUP BY 1 ), contact\_profile AS ( SELECT c.contact\_pk, c.contact\_name, -- Additional fields and complex COALESCE statements -- Truncated for brevity FROM \`ra-development.analytics.contacts\_dim\` c LEFT JOIN \`ra-development.analytics.contact\_companies\_fact\` cc ON c.contact\_pk = cc.contact\_fk AND contact\_next\_created\_date IS NULL LEFT JOIN contact\_conversations s ON c.contact\_name = s.contact\_name LEFT JOIN \`ra-development.analytics.companies\_dim\` d ON cc.company\_fk = d.company\_pk LEFT JOIN \`ra-development.analytics.contracts\_fact\` f ON c.contact\_pk = f.contact\_fk ), contact\_profiles\_deduped AS ( SELECT contact\_pk, contact\_name, company\_lifecycle\_stage, contact\_brief, company\_name, ideal\_customer\_group FROM contact\_profile GROUP BY 1,2,3,4,5,6 ), predictions as ( SELECT contact\_pk, contact\_name, contact\_brief, company\_name, ideal\_customer\_group, ml\_generate\_text\_result\['predictions'\]\[0\]\['content'\] AS predicted\_decision\_maker\_type, ml\_generate\_text\_status as prediction\_status, FROM ML.GENERATE\_TEXT( MODEL \`analytics\_ai.bison\_model\`, ( SELECT CONCAT('Classify this person as one of the following categories of probability of being a budget-holding decision maker in sales activity with us on behalf of their organisation,putting emphasis on their job title and whether they have signed statement of work or project delivery contracts before, (not employment, redundancy or related contracts) and with having no explicit job title reducing that probability, as either High,Medium or Low. Also predict their categories of consulting services interest from the list "Operational Reporting", "Marketing Analytics", "Data Centralization", "Product Analytics", "Marketing Attribution" or "Unknown", separating multiple categories if predicted for a contact using commas, and then predict the category of organization or company that the person works for from Startup, SMB, Mid-Market, Enterprise. Give me the rationale your categorisations ensuring any PII is obfuscated in that rationale and do not return any words or comments apart from these three sets of categorisations and rationale. Your response should be in the form of a single line of text that separates each categorisation for the person using a colon, for example: High:Marketing Analytics,Product Analytics:SMB:Signed statement of work contracts before and has contacted about operational reporting related topics. Text:',contact\_name,'"',contact\_brief,'"') AS prompt, \* FROM contact\_profiles\_deduped order by length(contact\_brief) desc limit 50 ), STRUCT( 0.0 AS temperature, 250 AS max\_output\_tokens)) ) select contact\_pk, contact\_name, contact\_brief, split(string(predictions.predicted\_decision\_maker\_type),':')\[safe\_offset(0)\] as decision\_maker\_rating, split(string(predictions.predicted\_decision\_maker\_type),':')\[safe\_offset(1)\] as predicted\_interest, split(string(predictions.predicted\_decision\_maker\_type),':')\[safe\_offset(2)\] as predicted\_company\_size, split(string(predictions.predicted\_decision\_maker\_type),':')\[safe\_offset(3)\] as rationale, coalesce(trim(prediction\_status),'OK') as prediction\_status from predictions Running this SQL query gave us predicted category segment values for (most of) the contacts within the scope of this query, with the `split` and `safe_offset()` string functions splitting out the various categorisation types from the LLM’s response, as you can see from the screenshot below. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752871916460-9crfog.png%3Fresize%3D3777%252C3243%26ssl%3D1) I say “most” of the contacts because some significant restrictions apply to using ML.GENERATE\_TEXT in set-based data transformations like these, the most significant being a [quota limit](https://cloud.google.com/bigquery/quotas) of 60 requests per minute that if exceeded return a RESOURCE EXCEEDED error message and no request response. Working around this limit therefore requires batching-up and sending requests using a shell script or another approach [such as the ones described here](https://www.googlecloudcommunity.com/gc/AI-ML/Running-batch-job-on-LLM/m-p/683677), not an impossible thing to do but it does effectively rule this feature out as a part of batch-style transformations in a dbt project, for example. The most likely scenario therefore is one where we batch-up these categorisations and output the results to a file or BigQuery table as a one-off or infrequently-scheduled process and then use the results to either create another table in the warehouse that we join-back to our contacts data, or use as the data source in a reverse ETL sync to HubSpot using a service such as Hightouch or Rudderstack Reverse ETL. ### Interested? Find Out More! Rittman Analytics is a boutique data analytics consultancy that helps ambitious, digital-native businesses scale-up their approach to data, analytics and AI. We’re authorised delivery partners for [Cube](https://www.rittmananalytics.com/partners/cube), [Dagster](https://www.rittmananalytics.com/partners/dagster), [Preset](https://www.rittmananalytics.com/partners/preset), [dbt Labs](https://www.rittmananalytics.com/partners/dbt-labs), [Fivetran](https://www.rittmananalytics.com/partners/fivetran), [Rudderstack](https://www.rittmananalytics.com/partners/rudderstack) and [Snowflake](https://www.rittmananalytics.com/partners/snowflake) along with [Google Cloud](https://www.rittmananalytics.com/partners/looker), [Oracle](https://www.rittmananalytics.com/partners/oracle), [Segment](https://www.rittmananalytics.com/partners/segment) and [Lightdash](https://www.rittmananalytics.com/partners/lightdash) and are experts at helping you choose the right ones for your organisation’s needs, use-cases and budget and working with you and your data team to successfully implement them. If you’re looking for some help and assistance with your AI initiative or would just like to talk shop and share ideas and thoughts on what’s going on in your organisation and the wider data analytics world, [contact us now](https://calendly.com/markrittman/initial-discovery-call-with-mark-rittman?utm_source=medium.com&utm_medium=blog&utm_campaign=customer_segmentation) to organise a 100%-free, no-obligation call — we’d love to hear from you! [AI](https://rittmananalytics.com/blog/tag/AI)[Google BigQuery](https://rittmananalytics.com/blog/tag/Google+BigQuery)[Marketing Analytics](https://rittmananalytics.com/blog/tag/Marketing+Analytics)[Generative AI](https://rittmananalytics.com/blog/tag/Generative+AI) [Mark Rittman](https://rittmananalytics.com/blog?author=5846da08f5e231d83261e221) CEO of Rittman Analytics, host of the Drill to Detail Podcast, ex-product manager and twice company founder. [https://rittmananalytics.com](https://rittmananalytics.com/) --- # Free Download : Best of the Rittman Analytics Blog 2023 eBook Canonical: https://rittmananalytics.com/blog/2023/12/29/2023-12-29-end-of-year-special-best-of-the-rittman-analytics-blog-2023-pdf-ebook URL: https://rittmananalytics.com/blog/2023/12/29/2023-12-29-end-of-year-special-best-of-the-rittman-analytics-blog-2023-pdf-ebook Description: So it’s almost the end of 2023 and it’s been another interesting year in the world of data analytics, the modern data stack and what surely must be the theme… Author: Mark Rittman Published: 2023-12-29T14:34:01+00:00 Modified: 2023-12-29T14:34:01+00:00 --- So it’s almost the end of 2023 and it’s been another interesting year in the world of data analytics, the modern data stack and what surely must be the theme of the year – artificial intelligence. Over the past twelve months we’ve written a number of articles for our blog on topics ranging from marketing analytics through to data orchestration, the new dbt Cloud CLI and of course generative AI and ChatGPT. And so to round-off 2023, we’ve collected-together the most popular posts from 2023 as a free PDF ebook that you can [download here](https://4402794.fs1.hubspotusercontent-na1.net/hubfs/4402794/ebooks/Best%20of%20the%20Rittman%20Analytics%20Blog%202023.pdf) – have a great New Year and see you all in 2024! --- # How Rittman Analytics Does Web Analytics and Marketing Attribution, using Google BigQuery, Looker, dbt and Segment — Rittman Analytics Canonical: https://rittmananalytics.com/blog/2023/12/08/how-rittman-analytics-does-web-analytics-and-marketing-attribution-using-google-bigquery-looker-dbt-and-segment-rittman-analytics URL: https://rittmananalytics.com/blog/2023/12/08/how-rittman-analytics-does-web-analytics-and-marketing-attribution-using-google-bigquery-looker-dbt-and-segment-rittman-analytics Description: How Rittman Analytics Does Web Analytics and Marketing Attribution, using Google BigQuery, Looker, dbt and Segment Marketing Analytics 8 Dec Written By Mark Rittman As a boutique data analytics consultancy that… Author: Mark Rittman Published: 2023-12-08T01:00:00+01:00 Modified: 2023-12-08T01:00:00+01:00 --- ![How Rittman Analytics Does Web Analytics and Marketing Attribution, using Google BigQuery, Looker, dbt and Segment — Rittman Analytics](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2023%2F12%2F1752871966417-xcev7r.png%3Ffit%3D2500%252C1653%26ssl%3D1) # How Rittman Analytics Does Web Analytics and Marketing Attribution, using Google BigQuery, Looker, dbt and Segment [Marketing Analytics](https://rittmananalytics.com/blog/category/Marketing+Analytics) 8 Dec Written By [Mark Rittman](https://rittmananalytics.com/blog?author=5846da08f5e231d83261e221) As a boutique data analytics consultancy that gets most of its new clients through enquiries from our website, its obviously fairly critical to our growth that potential new clients find and discover our site easily and once they do, stick around and (hopefully, [why don’t you try it out…](https://calendly.com/markrittman/scaling-analytics-discovery-call)) book a discovery call with us to find-out how we can make their analytics vision a reality. Looking at this in the form of a classic marketing funnel you can divide those marketing activities and the spend they incur into three broad categories: 1. “Top of Funnel” activities and touchpoints that attract traffic to our site that will find our content relevant and engaging 2. “Middle of Funnel” website pages and marketing touchpoints that help visitors with an interest in our services find the information they need 3. “Bottom of Funnel” site pages and third-party services that aim to turn that interest into a booked-meeting to find-out how we can help ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752871966417-xcev7r.png%3Fresize%3D7348%252C4859%26ssl%3D1) We therefore wanted to create a web performance dashboard that brought together all of our key web performance metrics in one place, organized by marketing funnel layer and adding enough context in order for those metrics to become actionable. As we did before with our [business KPIs dashboard](https://blog.rittmananalytics.com/kpi-dashboards-and-balanced-scorecards-using-looker-dbt-and-google-bigquery-f248d69d651a), the first step was to sit down with stakeholders and come-up with a set of problems to be solved and a process by which solve those problems, expressed in the form of a wireframe dashboard design shown in the diagram below. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752871966372-a0rma.png%3Fresize%3D3678%252C4143%26ssl%3D1) Creating such as dashboard that spanned all of our marketing activities, visitor digital touchpoints and data on their transaction history required us to bring data in from a number of different data sources including website clickstream data, lead collection and meeting booking data from third-party services such as [ConvertBox](https://convertbox.com/) and Calendly, ad network and social media data as well as operational stored in our centralised data warehouse, Google BigQuery. To do this we first landed all of our marketing and customer data sources in [Google BigQuery](https://rittmananalytics.com/partners/looker), our cloud data warehouse and then used the open-source [dbt Core](https://rittmananalytics.com/partners/dbt-labs) toolkit to format, join and then combine all of this data into a single dataset and “version of the truth” ready for building our dashboard in [Looker](https://rittmananalytics.com/partners/looker). ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752871966467-r8v5ia.png%3Fresize%3D3842%252C2936%26ssl%3D1) (We could, theoretically at least, have use Looker Studio (or even regular Looker) to somewhat shortcut this process to instead create a “single pane of glass” integrated dashboard view of these data sources, but this would quickly become unwieldy as the number of data sources increased and would still would leave us with silos of reporting data that could only be analyzed in isolation, not connected to each other) The Web Performance and Marketing Attribution dashboard is shown in the screenshot below, where we’ve taken the wireframe design and used it as the design template for a Looker dashboard. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752871966560-24rfd.png%3Fresize%3D4420%252C4312%26ssl%3D1) The dashboard is organized into four sections, one for headline metrics and three others that correspond to (groups of) layers in our marketing funnel. 1. Headline Metrics & KPIs : key metrics and indicators that, at-a-glance, give us the state of our marketing activities over the past 30 days 2. Top-of-Funnel Metrics & KPIs : metrics and data visualizations that tell us how we’ve performed at bringing-in high-quality traffic to our site 3. Middle-of-Funnel Metrics & KPIs : metrics and KPIs that measure how effective we’ve been at directing and helping that traffic discover relevant content about the services we offer 4. Bottom-of-Funnel Metrics & KPIs : metrics and visualizations that show how each of our marketing channels have contributed to conversions (booked meetings) and goals achieved (contact buttons pressed) #### Headline Conversion Metrics and Identity Stitching For our headline metrics and alongside obvious ones to include such as Sessions, Users, Paid Ad Spend and Cost-per-Click (CPC), we also wanted to highlight performance in bringing-in first meetings with potential new clients as those are the biggest driver of new incremental revenue for our business. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752871966441-f0hwge.png%3Fresize%3D2224%252C1342%26ssl%3D1) To emphasise the value of these first meetings we [identity-stitched](https://www.rittmananalytics.com/blog/2022/5/30/stitching-identity-across-the-customer-journey-using-segment-google-bigquery-and-looker) the anonymous and identified website interactions for each of our converting users, making sure that at all times when we hand-off a user to a third-party service such as Calendly or ConvertBox we always include the device-based anonymousId that Segment assigns to each site visitor. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752871966471-o5p7j.png%3Fresize%3D1864%252C1330%26ssl%3D1) Doing so ensures that when we send Segment track and identify events from those services back to Segment to record a conversion, we can then connect those events back to the journey that user took anonymously that eventually led to that conversion. Because we identify our site users using their email address, we can also then connect those users to data from HubSpot, our sales and CRM system that feeds into our operational data warehouse, allowing us to assign first conversion and lifetime value revenue numbers to those converting users albeit with several weeks or months lag between conversion and deal closure. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752871966496-ielbf.png%3Fresize%3D3842%252C1276%26ssl%3D1) #### Measuring Middle-of-Funnel Performance for acquisition channels and marketing pages For each of the marketing channels that bring visitors to our website along with blog articles and the marketing, services and case-study pages on our site, our measurement of performance is how effective each were in moving those visitors down through our marketing funnel. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752871966281-dmr75p.png%3Fresize%3D3240%252C2648%26ssl%3D1) Data visualisations such as these that connect-together activity at one stage in the marketing funnel — interacting with a post we published on LinkedIn, liking an X tweet or clicking on a paid social promotion, for example — with subsequent session page view data from the website sessions resulting from those interactions — are easy to set-up in Looker using the Merge Query feature, as you can see from the example below. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752871966532-3shdub.png%3Fresize%3D4516%252C2787%26ssl%3D1) #### Attributing Conversions to our Top-, Middle- and Bottom-of-Funnel Marketing Channels For the final, bottom-of-funnel channel layer of our Web Performance dashboard, reporting on conversions in a meaningful way is surprisingly hard: - Do we report the total number of conversions (meeting bookings) over a given time period and compare that to the total number of user sessions? Or does session conversion rate (total sessions containing at least one conversion divided by total sessions) or even user conversion rate (number of converting users compared to total users)? - Are we interested in all conversions by a user, or do we wish to report separately on first conversions (far more important, revenue-wise) and then roll-up any second, third, fourth conversions into a single repeat conversions metric? - Most importantly though in determining the value each channel delivers for our business, how do we meaningfully attribute that value when last-click attribution (assigning the conversion to the channel the conversion event occurred in) assigns that conversion in most cases to direct. For users that go on to book meetings its more typical that they arrive by finding us in a Google Search, take a look around and then return a few days later to book a meeting to find out more about our services.  That same user might not be in a position to move ahead right then with their project but returns a week later after being reminded of us through a paid social post on LinkedIn, gets distracted and then finally types our website address into their browser, clicks on the button to book a meeting and converts for a second time. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752871966408-ij4zox.png%3Fresize%3D2442%252C2336%26ssl%3D1) What we therefore need is a method of attributing conversions to marketing channels that enables us to thereby understand: - which marketing channels are most effective in bringing-in users that eventually go-on to convert (first non-direct click attribution) - which marketing channels are most effective in closing that interest and turning it into a conversion (last non-direct click attribution) - what are the mix of marketing channels that typically feature in a converting visitor journey (even-click or linear attribution) We do this by leveraging the [multi-touch, multi-cycle marketing attribution dbt package](https://github.com/rittmananalytics/ra_attribution) we’ve made available on our public Github account (along with a [Looker block version optimized for Google Analytics 4](https://github.com/rittmananalytics/ra_attribution_for_ga4)) and used it, together with the identity-stitching logic outlined earlier in this blog to attribute the value of each users’ conversion across each separate conversion cycle, as you can see in the recent example shown below. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752871966519-dom3ko.png%3Fresize%3D6142%252C2764%26ssl%3D1) We then use the first non-direct click, last non-direct click and even (linear)-click attribution model data to provide visualizations for the final marketing funnel layer of our dashboard,  ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752871966485-3do52c.png%3Fresize%3D3482%252C1676%26ssl%3D1) ### Interested? Find Out More! Rittman Analytics is a boutique data analytics consultancy that works with growth-stage, mid-market and enterprise businesses in the UK, EU and North America to help level-up their marketing analytics capabilities and build world-class web performance and marketing attribution models – just like the one we’ve built for ourselves and described in this blog post. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752871966444-i4esnj.png%3Fresize%3D1588%252C530%26ssl%3D1) We’re authorised delivery partners for [Google Cloud](https://www.rittmananalytics.com/partners/looker) (for Looker and Looker Studio), [Snowflake](https://www.rittmananalytics.com/partners/snowflake), [Oracle](https://www.rittmananalytics.com/partners/oracle), [Segment](https://www.rittmananalytics.com/partners/segment) and [Rudderstack](https://www.rittmananalytics.com/partners/rudderstack) along with [Cube](https://www.rittmananalytics.com/partners/cube), [Dagster](https://www.rittmananalytics.com/partners/dagster), [Preset](https://www.rittmananalytics.com/partners/preset), [dbt Labs](https://www.rittmananalytics.com/partners/dbt-labs), [Fivetran](https://www.rittmananalytics.com/partners/fivetran) and [Lightdash](https://www.rittmananalytics.com/partners/lightdash) — we love them all and are experts at helping you choose the right ones for your organisation’s needs, use-cases and budget and working with you and your data team to make them all a success. If you’re looking for some help and assistance with your marketing analytics or would just like to talk shop and share ideas and thoughts on what’s going on in your organisation and the wider data analytics world, [contact us now](https://calendly.com/markrittman/initial-discovery-call-with-mark-rittman?utm_source=medium.com&utm_medium=blog&utm_campaign=customer_segmentation) to organise a 100%-free, no-obligation call — we’d love to hear from you! [Marketing Analytics](https://rittmananalytics.com/blog/tag/Marketing+Analytics)[Tips & Techniques](https://rittmananalytics.com/blog/tag/Tips+%26+Techniques)[Segment](https://rittmananalytics.com/blog/tag/Segment)[Looker](https://rittmananalytics.com/blog/tag/Looker)[dbt](https://rittmananalytics.com/blog/tag/dbt)[Google BigQuery](https://rittmananalytics.com/blog/tag/Google+BigQuery)[Marketing Attribution](https://rittmananalytics.com/blog/tag/Marketing+Attribution) [Mark Rittman](https://rittmananalytics.com/blog?author=5846da08f5e231d83261e221) CEO of Rittman Analytics, host of the Drill to Detail Podcast, ex-product manager and twice company founder. [https://rittmananalytics.com](https://rittmananalytics.com/) --- # How Rittman Analytics Builds Data Stacks for Growth-Stage Businesses using Cube, Dagster and Preset — Rittman Analytics Canonical: https://rittmananalytics.com/blog/2023/11/12/how-rittman-analytics-builds-data-stacks-for-growth-stage-businesses-using-cube-dagster-and-preset-rittman-analytics URL: https://rittmananalytics.com/blog/2023/11/12/how-rittman-analytics-builds-data-stacks-for-growth-stage-businesses-using-cube-dagster-and-preset-rittman-analytics Description: How Rittman Analytics Builds Data Stacks for Growth-Stage Businesses using Cube, Dagster and Preset Modern Data Stack 12 Nov Written By Mark Rittman One of the new client case studies… Author: Mark Rittman Published: 2023-11-12T01:00:00+01:00 Modified: 2023-11-12T01:00:00+01:00 --- ![How Rittman Analytics Builds Data Stacks for Growth-Stage Businesses using Cube, Dagster and Preset — Rittman Analytics](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2023%2F11%2F1752872008432-roe86.png%3Ffit%3D1518%252C1252%26ssl%3D1) # How Rittman Analytics Builds Data Stacks for Growth-Stage Businesses using Cube, Dagster and Preset [Modern Data Stack](https://rittmananalytics.com/blog/category/Modern+Data+Stack) 12 Nov Written By [Mark Rittman](https://rittmananalytics.com/blog?author=5846da08f5e231d83261e221) One of the [new client case studies we published](https://www.rittmananalytics.com/case-studies/ospreycharging) earlier this month was for our work with [Osprey Charging Networks](https://www.ospreycharging.co.uk/), one of the [most-popular and fastest-growing](https://www.zap-map.com/news/drivers-rate-electric-vehicle-charging-networks-uk) EV charging networks in the UK and based not far from us in London, England. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872008432-roe86.png%3Fresize%3D1518%252C1252%26ssl%3D1) As Lewis and I are both EV owners we know the value of a fast, reliable charging network and [with Osprey planning to install over 800 new chargers in 2023](https://www.rittmananalytics.com/case-studies/ospreycharging), we’ve been working with their team to modernise and scale their data stack and data team to ensure it supports and powers Osprey’s growth over years to come. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872008197-q8pk4a.png%3Fresize%3D4941%252C2064%26ssl%3D1) In the past the choice of components for such a data stack would have been limited but as vendors such as Google Cloud (for Looker) and dbt Labs increasingly focus on enterprise, rather than growth businesses and as monolithic BI suites become increasingly disaggregated, we saw an opportunity to recommend a data stack architecture that was both best-of-breed and worked for startups in today’s more challenging funding environment. - Snowflake as the cloud data warehouse, a safe and easy choice - dbt Core (but not dbt Cloud) for data modeling and transformation - Dagster Cloud for data orchestration and data asset lineage - Cube Cloud for the semantic model, caching and role-based access - Preset Cloud, the commercial version of Superset, for dashboards and reports - Fivetran for data pipelines and other data ingestion ### The Growth-Stage Modern Data Stack The choice of vendors we recommended for Osprey Charging’s data stack is one example of the vendor partnerships and data stack component recommendations we’ve made this year for growth-stage organisations who we’ve found have broadly similar needs and goals: - A desire for a Looker-style dashboarding tool with an integrated business semantic layer - Data assets and a data architecture that extend beyond just batch SQL transformations updating warehouse tables on an hourly schedule - A budget for their data initiative that two years ago might well have been north of £250k — £500k but now is likely to be just half or a quarter of that, including spend on getting someone in like ourselves to get the first stage delivered Generalised beyond just Osprey Charging and with further options such as [Motherduck](https://motherduck.com/) for the cloud data warehouse, [Rudderstack](https://www.rudderstack.com/) for adding event collection, [Airbyte](https://airbyte.com/) for as an alternative to Fivetran for batch data extract and [Estuary](https://estuary.dev/) for ingesting data in real-time as well as services such as [Rudderstack](https://www.rudderstack.com/product/reverse-etl/) and [Hightouch](https://hightouch.com/) for data activation and reverse ETL, the diagram below shows the growth-stage data stack layers and services we generally recommend for clients today. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872008411-obm2t.png%3Fresize%3D3987%252C4308%26ssl%3D1) ### The Enabler: Cube, Dagster and Preset Integration There have, of course, always been niche vendors within the modern data stack ecosystem but what makes this type of architecture increasingly viable is the integration between Cube, Dagster and Preset that’s now available as features provided by the commercial cloud-hosted versions of those products. Cube, the disaggregated or “headless BI” semantic layer that also provides role-based access control and tools to pre-aggregate query results in your cloud data warehouse now comes with a [Semantic Layer Sync](https://cube.dev/blog/introducing-semantic-layer-sync) feature in its cloud version that automatically creates, and then keeps in-sync, dataset definitions in tools such as Preset, Metabase and (coming soon) Power BI. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872008490-wunjsb.png%3Fresize%3D1400%252C889%26ssl%3D1) Another one of our clients, [Breakthrough](https://rittmananalytics.com/case-studies/breakthrough), was an early adopter of Cube’s Semantic Layer Sync and found value in [having their BI tool work off the same semantic model as the data products that they embedded Cube into](https://cube.dev/blog/introducing-semantic-layer-sync) as their original use-case for their technology. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872008401-0hlloh.png%3Fresize%3D1600%252C898%26ssl%3D1) Both Cube and Dagster now have the ability to connect to dbt (via the dbt manifest file or via your dbt package’s git repository) to [ingest model definitions](https://cube.dev/blog/introducing-dbt-integration-with-cube) and the [entire dbt transformation DAG](https://docs.dagster.io/integrations/dbt/using-dbt-with-dagster/load-dbt-models) respectively into their own repositories, which coupled with [Dagster’s ability to trigger builds of Cube pre-aggregations](https://cube.dev/docs/product/apis-integrations/orchestration-api/dagster) using Cube’s Orchestration API gives you a level of component integration as deep and in-fact wider than you had previously with Looker. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872008647-ukcw7s.png%3Fresize%3D3015%252C1758%26ssl%3D1) Preset is a great ad-hoc query tool and with this new, closer integration with Cube’s semantic layer the user experience is like Tableau connected to a Looker-style semantic model, something that’s possible now with Preset and Cube but still not available with actual Tableau and Looker Modeler at the time of writing [despite being announced](https://cloud.google.com/blog/products/data-analytics/introducing-looker-modeler) more than six months ago. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872008353-qgxuxd.png%3Fresize%3D1466%252C1078%26ssl%3D1) My colleagues Jordan Ilyat and Olivier Dupuis have a number of articles on Dagster including one on [orchestrating Fivetran data extracts using Dagster](https://blog.rittmananalytics.com/unlocking-data-orchestration-how-to-orchestrate-fivetran-and-dbt-with-dagster-eb03eb973dee) and [configuring Dagster to refresh data items in your warehouse based on freshness policies](https://blog.rittmananalytics.com/optimizing-data-materialization-using-dagsters-policies-308fc77aba3c) rather than a schedule, potentially saving you considerable running costs for your warehouse orchestration now that both Dagster and dbt Cloud have moved to consumption-based pricing. The real advantage with Dagster comes when you have more than just dbt jobs to orchestrate to refresh data in your data stack, as in-fact most organisations do once you factor-in data extraction and data activation into your overall data architecture and as Olivier does in his Medium article. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872008419-52qs.png%3Fresize%3D1400%252C1296%26ssl%3D1) ### How We Build Growth-Stage Data Stacks Finally, if you’re interested in reading and hearing more about our approach to growth data stacks using Dagster, Cube and Preset then be sure to checkout two webinars on this topic that ran earlier this year: - “[How Rittman Analytics Builds Modern Data Stacks using Cube, Preset and Dagster](https://www.youtube.com/watch?v=6Q8vGTvSar4)” where we cover all three tools along with a tool we use internally for automatically creating Cube metadata [https://www.youtube.com/embed/6Q8vGTvSar4?feature=oembed](https://www.youtube.com/embed/6Q8vGTvSar4?feature=oembed)","url":"[https://www.youtube.com/watch?v=6Q8vGTvSar4"](https://www.youtube.com/watch?v=6Q8vGTvSar4");,"width":200,"height":113,"providerName":"YouTube","thumbnailUrl":"[https://i.ytimg.com/vi/6Q8vGTvSar4/hqdefault.jpg"](https://i.ytimg.com/vi/6Q8vGTvSar4/hqdefault.jpg");,"resolvedBy":"youtube"}” data-block-type=”32″ id=”block-yui\_3\_17\_2\_1\_1699830500068\_97666″\> - “[How Rittman Analytics delivers the semantic layer today with Cube](https://www.youtube.com/watch?v=EWZmQlCvX3M)” that we ran with Cube’s partner team that focused specifically on our Cube development process. [https://www.youtube.com/embed/EWZmQlCvX3M?feature=oembed](https://www.youtube.com/embed/EWZmQlCvX3M?feature=oembed)","url":"[https://www.youtube.com/watch?v=EWZmQlCvX3M"](https://www.youtube.com/watch?v=EWZmQlCvX3M");,"width":200,"height":113,"providerName":"YouTube","thumbnailUrl":"[https://i.ytimg.com/vi/EWZmQlCvX3M/hqdefault.jpg"](https://i.ytimg.com/vi/EWZmQlCvX3M/hqdefault.jpg");,"resolvedBy":"youtube"}” data-block-type=”32″ id=”block-yui\_3\_17\_2\_1\_1699830500068\_111807″\> We still love Looker and as a [Google Cloud Partner](https://cloud.google.com/find-a-partner/partner/rittman-analytics) we’re super-excited to see [Looker Modeler](https://cloud.google.com/blog/products/data-analytics/introducing-looker-modeler) when it becomes generally available and in the meantime we’re evaluating the new [Open SQL Interface for Looker](https://cloud.google.com/looker/docs/sql-interface) that makes its semantic model accessible to other tools via its own JDBC driver. But in the meantime there’s now an alternative in the form of Cube, Dagster and Preset that arguably has more features, is just as integrated and can cost a lot less to run — leaving you money in your budget to have us come in and do for your organisation what we did for Osprey Charging Network. ### Interested? Find Out More! Rittman Analytics is a boutique data analytics consultancy that works with growth-stage, mid-market and enterprise businesses in the UK, EU and North America who want to build and enable a data team based around a modular, scalable modern data stack. We’re authorised delivery partners for [Cube](https://www.rittmananalytics.com/partners/cube), [Dagster](https://www.rittmananalytics.com/partners/dagster), [Preset](https://www.rittmananalytics.com/partners/preset), [dbt Labs](https://www.rittmananalytics.com/partners/dbt-labs), [Fivetran](https://www.rittmananalytics.com/partners/fivetran), [Rudderstack](https://www.rittmananalytics.com/partners/rudderstack) and [Snowflake](https://www.rittmananalytics.com/partners/snowflake) along with [Google Cloud](https://www.rittmananalytics.com/partners/looker) (for Looker and Looker Studio), [Oracle](https://www.rittmananalytics.com/partners/oracle), [Segment](https://www.rittmananalytics.com/partners/segment) and [Lightdash](https://www.rittmananalytics.com/partners/lightdash) — we love them all and are experts at helping you choose the right ones for your organisation’s needs, use-cases and budget and working with you and your data team to successfully implement them. If you’re looking for some help and assistance with your data project or would just like to talk shop and share ideas and thoughts on what’s going on in your organisation and the wider data analytics world, [contact us now](https://calendly.com/markrittman/initial-discovery-call-with-mark-rittman?utm_source=medium.com&utm_medium=blog&utm_campaign=customer_segmentation) to organise a 100%-free, no-obligation call — we’d love to hear from you! [Dagster](https://rittmananalytics.com/blog/tag/Dagster)[Cube](https://rittmananalytics.com/blog/tag/Cube)[Preset](https://rittmananalytics.com/blog/tag/Preset)[Modern Data Stack](https://rittmananalytics.com/blog/tag/Modern+Data+Stack) [Mark Rittman](https://rittmananalytics.com/blog?author=5846da08f5e231d83261e221) CEO of Rittman Analytics, host of the Drill to Detail Podcast, ex-product manager and twice company founder. [https://rittmananalytics.com](https://rittmananalytics.com/) --- # Coalesce 2023 and the new dbt Cloud CLI Canonical: https://rittmananalytics.com/blog/2023/10/29/coalesce-2023-and-the-new-dbt-cloud-cli URL: https://rittmananalytics.com/blog/2023/10/29/coalesce-2023-and-the-new-dbt-cloud-cli Description: “5. Expect to see GCP moving increasingly into packaged SaaS application and analytics solutions for ERP, CRM and Financials to complement their commoditised IaaS and PaaS cloud business and leveraging their massive GSuite and increasingly ChromeOS install base … and a business model that could provide these applications and packaged analytics for free… That’s the real existential threat to Oracle; spending all their time trying to win an un-winnable cloud infrastructure war and then GCP coming along and making ERP, CRM and business applications and their analytics essentially free.” and [on Twitter](https://twitter.com/markrittman/status/1136667379499511808?ref_src=twsrc%5Etfw%7Ctwcamp%5Etweetembed%7Ctwterm%5E1136667379499511808%7Ctwgr%5E4bade0df022ed12c8d0b21820d0625ea42a89843%7Ctwcon%5Es1_c10&ref_url=https%3A%2F%2Fpublish.twitter.com%2F%3Fquery%3Dhttps3A2F2Ftwitter.com2Fmarkrittman2Fstatus2F1136667379499511808widget%3DTweet): [https://platform.twitter.com/widgets.js](https://platform.twitter.com/widgets.js)","width":550,"height":null,"resolvedBy":"twitter","providerName":"Twitter"}” data-block-type=”22″ id=”block-yui\_3\_17\_2\_1\_1703337665056\_8869″\> > 2/5 Ever since Thomas Kurian joined @GCPcloud it was obvious he’d want to buy a full-featured analytics platform; look-back at the acquisition of Siebel Analytics in the mid-90s, with Looker 5 being GCP’s OBIEE10g and Data Studio being today’s Discoverer > > — Mark Rittman (@markrittman) [June 6, 2019](https://twitter.com/markrittman/status/1136667379499511808?ref_src=twsrc%5Etfw) [Pre-built analytics solutions for Oracle](https://www.oracle.com/business-analytics/business-intelligence/technologies/bi-applications/) and other vendors’ ERP, CRM and Financials applications, running on-top of Oracle’s suite of database and analytics tools, were always super-popular with Oracle’s customers as — in theory at least — the hard work required to build an integrated data warehouse and set of dashboards had already been done for you, bringing down the time to deploy an enterprise reporting solution from months to weeks. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872045634-q16e4j.png%3Fresize%3D1220%252C754%26ssl%3D1) The value delivered to customers using this buy vs. build approach to enterprise data warehousing was such that the license cost of each individual application — Sales Analytics, Financial Analytics, Customer Analytics and so on — were typically priced at 2–3x the price of the analytics tools they ran on. People buy solutions, not technology, as the saying goes, and packaged analytics solutions built on pre-built and integrated data warehouses were (and still are) a key part of Oracle’s appeal to the enterprise market — the market that Google had hired Thomas Kurian in help them break-into. And if you’re a long-term reader of this blog you’ll probably be aware of Rittman Analytics’ take on packaged analytics and data warehousing, our dbt package and open-sourced warehousing toolkit called [RA Warehouse for dbt](https://github.com/rittmananalytics/ra_data_warehouse). It uses a similar design approach to Oracle BI Applications’ packaged data warehouse but focuses instead on SaaS application sources such as Hubspot, Xero and Jira, uses updated modern data stack technology such as Google BigQuery, dbt and Looker and creates a single integrated view of your business across key subject areas. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872045629-3msa6e.png%3Fresize%3D1400%252C806%26ssl%3D1) So how does Google Cloud Cortex Data Foundation compare to these packaged analytics approaches, and how can our RA Data Warehouse and other partner data sources and BI tools integrate with and extend Google’s framework? Google Cloud Cortex Foundation, as you’d expect from an initiative designed to drive usage and consumption of Google Cloud Platform services, uses Google Cloud Platform tools and services to enable data integration, processing, analytics and reporting. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872045557-ys9v4w.png%3Fresize%3D1246%252C721%26ssl%3D1) - Google BigQuery is used as the main data warehouse to store raw replicated data, processed change data capture tables and reporting datasets generated from analytical models. - Google Cloud Storage buckets are used to store files generated during deployment like DAG scripts, SQL queries and temporary processing files. - Google Cloud Build is used to trigger deployment processes that execute steps to deploy datasets, views, models and other artifacts to BigQuery. - Google Cloud Composer, rather that dbt or more suprisingly, [Dataform](https://cloud.google.com/dataform), is the data integration and orchestration tool used to transform data and schedule change data capture processing through custom DAGs - Google Cloud Dataflow is used as the data extraction and data pipeline tool to replicate data from Google Ads into BigQuery, for example - Google Secret Manager stores credentials needed to connect deployment scripts at runtime to data sources such as Salesforce - Identity and Access Management provisions and controlls access to resources through roles, permissions and service accounts At this point in time, the supported data sources for Cortex Data Foundation are SAP, Salesforce and a number of marketing sources. Each data source has its own method of data extraction, for example: - SAP ECC or SAP S/4HANA transactional data can be streamed into BigQuery using partner tools such as SnapLogic, Boomi, Informatica Cloud etc; if real-time replication of SAP data is not required, scheduled exports from SAP can be loaded into BigQuery tables on a periodic basis with change data capture (CDC) processing then applied to surface only the latest version in separate CDC datasets - For Salesforce data, the deployment process sets up a connected app in Salesforce for authentication, and then raw data is then extracted from Salesforce APIs in real-time with CDC processing updating the latest records. - Google Ads data is integrated using the Google Cloud Dataflow runner to ingest raw campaign performance data from Google Ads APIs. - Google Campaign Management 360 (CM360) and other platforms such as TikTok also have replication modules that extract raw APIs/exports into BigQuery for further processing. Some degree of data centralization and integration across these sources is provided by the framework into what’s termed the “K9” (as in DAGs…) datsets but not to the point where single deduplicated customer records, for example, are created. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872045808-djxyf9.png%3Fresize%3D1127%252C701%26ssl%3D1) But Cortex Data Foundation is part of an open framework and Fivetran, for example, can be used to replicate on-premises database sources such as SAP HANA into BigQuery using Fivetran High Volume Agent, as shown in the diagram before. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872045625-tm8cyd.png%3Fresize%3D1400%252C501%26ssl%3D1) Or you can use Fivetran to do what we’re currently working on as an extension to Cloud Cortex Framework; integrating our [RA Warehouse data centralization framework](https://github.com/rittmananalytics/ra_data_warehouse) and [Fivetran’s connectors](https://www.fivetran.com/connectors) into Cortex Data Foundation to bring in the long tail of other data sources such as Hubspot, Xero, Netsuite and Oracle Fusion HCM and CRM … and create a single view of customers, companies, products and other key business entities. ### INTERESTED? FIND OUT MORE Rittman Analytics is a boutique analytics consultancy specializing Google Cloud Platform and the modern data stack, who can [centralise your data sources](https://markr320.sg-host.com/data-centralization), [optimize your marketing activity](https://markr320.sg-host.com/#:~:text=OUR%20SOLUTIONS-,Marketing%20Attribution,-Modern%20Data%20Stack) and [enable your end-users and data team](https://markr320.sg-host.com/data-team-enablement) with best practices and a [modern analytics workflow](https://markr320.sg-host.com/getting-started-with-dbt). If you’re looking for some help and assistance getting started with Google Cloud Cortex Framework or to help build-out your analytics capabilities and data team using a modern, flexible and modular data stack, [contact us now](https://calendly.com/markrittman/initial-discovery-call-with-mark-rittman?utm_source=medium.com&utm_medium=blog&utm_campaign=customer_segmentation) to organize a 100%-free, no-obligation call — we’d love to hear from you! [Data Engineering](https://rittmananalytics.com/blog/tag/Data+Engineering)[Google Cloud Platform](https://rittmananalytics.com/blog/tag/Google+Cloud+Platform)[Google BigQuery](https://rittmananalytics.com/blog/tag/Google+BigQuery)[Oracle](https://rittmananalytics.com/blog/tag/Oracle) [Mark Rittman](https://rittmananalytics.com/blog?author=5846da08f5e231d83261e221) CEO of Rittman Analytics, host of the Drill to Detail Podcast, ex-product manager and twice company founder. [https://rittmananalytics.com](https://rittmananalytics.com/) --- # Bringing dbt and Analytics Engineering to Oracle Autonomous Data Warehouse — Rittman Analytics Canonical: https://rittmananalytics.com/blog/2023/09/07/bringing-dbt-and-analytics-engineering-to-oracle-autonomous-data-warehouse-rittman-analytics URL: https://rittmananalytics.com/blog/2023/09/07/bringing-dbt-and-analytics-engineering-to-oracle-autonomous-data-warehouse-rittman-analytics Description: Bringing dbt and Analytics Engineering to Oracle Autonomous Data Warehouse OracleAnalytics Engineering 7 Sept Written By Mark Rittman dbt (Data Build Tool) is an open source command-line tool for managing… Author: Mark Rittman Published: 2023-09-07T01:00:00+01:00 Modified: 2023-09-07T01:00:00+01:00 --- ![Bringing dbt and Analytics Engineering to Oracle Autonomous Data Warehouse — Rittman Analytics](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2023%2F09%2F1752872081516-7xugg7.jpg%3Ffit%3D2048%252C776%26ssl%3D1) # Bringing dbt and Analytics Engineering to Oracle Autonomous Data Warehouse [Oracle](https://rittmananalytics.com/blog/category/Oracle)[Analytics Engineering](https://rittmananalytics.com/blog/category/Analytics+Engineering) 7 Sept Written By [Mark Rittman](https://rittmananalytics.com/blog?author=5846da08f5e231d83261e221) [dbt (Data Build Tool)](https://www.getdbt.com/blog/what-exactly-is-dbt/) is an open source command-line tool for managing and automating data transformation workflows. It enables a standardised, modular approach to building analytics code and is typically used as part of an increasingly popular method of developing data warehouses and data platforms called [analytics engineering](https://www.getdbt.com/what-is-analytics-engineering/). The analytics engineering approach combines software engineering principles with analytics to create more maintainable, trustworthy data pipelines built on transformation logic that is modular, scalable, tested, and well-documented. With dbt, analytics code progresses from messy, ad hoc scripts to structured engineering workflows developed using a [software development lifecycle](https://www.airops.com/blog/5-reasons-dbt-sets-the-standard-for-data-transformation-tools). In this post, I’ll introduce dbt, explain its benefits for Oracle Database and Autonomous Data Warehouse developers and walk through getting started with dbt and the new [dbt-oracle adapter](https://github.com/oracle/dbt-oracle). #### What is dbt? ELT (Extract, Load, Transform) is a modern data warehousing pattern very familiar to Oracle data warehouse developers where raw data is first extracted and loaded into the target warehouse, then transformed within the warehouse itself. This is where dbt comes in — it provides the “T” for managing transformations in ELT, with the E (extract) and L (load) typically handled by services such as [Fivetran](https://www.fivetran.com/), Airbyte or Stitch. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872081516-7xugg7.jpg%3Fresize%3D2048%252C776%26ssl%3D1) Originally developed to help data analysts within the VC-backed startup world develop production-ready data transformation code for Amazon Redshift and other cloud data warehouses, dbt is now more of an ecosystem than single tool and comprises of: - An open source core ([dbt Core](https://pypi.org/project/dbt-core/)), originally developed by Fishtown Analytics (now dbt Labs) that you run from the command-line and can schedule using cron jobs or orchestrators such as [Dagster](https://docs.dagster.io/integrations/dbt) - a commercial SaaS service ([dbt Cloud](https://www.getdbt.com/product/dbt-cloud/)) that provides features for developing, testing, scheduling and investigating data models - community-provided utilities and content packages (such as our [RA Attribution](https://github.com/rittmananalytics/ra_attribution) package for multi-touch marketing attribution) along with [official](http://getdbt.slack.com/) and [related](http://locallyoptimistic.slack.com/) Slack groups Key capabilities of dbt include: - Dependency management — dbt handles model dependencies automatically - Testing — add tests to models to ensure data quality - Refactoring — easily refactor models without breaking dependencies - Documentation — auto-generate docs for all models in a project - CI/CD Integration — build, test, and deploy projects on every code change Whereas tools such as Oracle Data Integrator (ODI) and Oracle Warehouse Builder provide a visual, GUI-based mapping interface for designing data flows, dbt instead takes a code-centric approach with transformations encapsulated in modular SQL SELECT statements called “models”, either written and run using a text editor or tool such as Microsoft VS Code. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872081701-iwj2p7.jpg%3Fresize%3D2896%252C1760%26ssl%3D1) Developing transformation logic using a GUI has the benefit of opening-up this task to a wider audience, but when the business requires complex data transformations such an approach can become unwieldy. More of a concern though is the ungoverned shadow SQL or Python code that drag-and-drop tool users end-up adding to their visual transformation nodes, making it hard to track changes and ensure those transformations and embedded are tested and reliable in a scalable and automated way. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872081826-ug0skx.jpg%3Fresize%3D1400%252C906%26ssl%3D1) A key difference between ODI and dbt is therefore in its approach to modularity and reusable transformation logic. With ODI, code reuse is achieved by copying and modifying mappings across projects; dbt, however achieves this through the use of those modular, configurable SQL models that can be packaged up into shared libraries and invoked repeatedly. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872081591-znq3x.jpg%3Fresize%3D1794%252C1402%26ssl%3D1) Also, crucially, dbt models exist as simple SQL files making them straightforward to version control via Git. ODI’s visual mappings are stored within its repository, requiring use of ODI lifecycle tools. With dbt, developers can utilize the full power of Git for branching strategies, pull requests, and decentralized development. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872081585-v7r8f.jpg%3Fresize%3D1400%252C904%26ssl%3D1) dbt models also provide visibility into history and differences between versions. #### Getting Started with dbt-oracle The [dbt-oracle adapter](https://github.com/oracle/dbt-oracle) (originally created by Indicium, now supported and maintained directly by Oracle) is a python package that contains the core dbt libraries together with an adapter for connecting to on-premises and cloud Oracle databases. This, along with an install of Python 3 and the [oracledb client drivers package](https://oracle.github.io/python-oracledb/) are what’s required to install dbt on your Mac, for example. python3 -m venv dbt-oracle-venv source dbt-oracle-venv/bin/activate pip install –upgrade pip python -m pip install oracledb –upgrade python -m pip install dbt-oracle The dbt-oracle Github repo at [https://github.com/oracle/dbt-oracle](https://github.com/oracle/dbt-oracle) contains a demo dbt package that transforms data from the SH (Sales History) example schema shipped with all Oracle database versions. Looking at an example dbt model within that demo package, you can see examples of model configuration settings, references to other tables in the form of sources and some Jinja templating code that adds a further WHERE clause to the SQL statement when the model is compiled in incremental load mode. {{config(materialized='incremental', unique\_key='group\_id')}} WITH direct\_sales\_promo\_cost AS ( SELECT s.prod\_id, s.quantity\_sold, s.amount\_sold, s.time\_id, c.channel\_desc, p.promo\_name, p.promo\_cost, {{ hash\_arguments(\['s.prod\_id', 's.quantity\_sold', 's.time\_id', 'p.promo\_name'\]) }} AS group\_id FROM {{ source('sh\_database', 'sales') }} s, {{ source('sh\_database', 'promotions') }} p, {{ source('sh\_database', 'channels') }} c WHERE s.channel\_id = 3 AND s.promo\_id = p.promo\_id AND s.channel\_id = c.channel\_id {% if is\_incremental() %} AND s.time\_id \> (SELECT MAX(time\_id) FROM {{ this }}) {% endif %} ) SELECT \* FROM direct\_sales\_promo\_cost Other model definitions use [ref() Jinja functions](https://docs.getdbt.com/reference/dbt-jinja-functions/ref) instead of table names to define dependencies between tables in a transformation DAG, and use Oracle-specific configuration settings to tell dbt how specifically to materialize the database object that is the result of compiling this particular model. {{config( materialized='table', parallel=4, table\_compression\_clause='COLUMN STORE COMPRESS FOR QUERY' ) }} select c.cust\_id, c.cust\_first\_name, c.cust\_last\_name, t.country\_iso\_code, t.country\_name, t.country\_region from {{ ref('sales\_internet\_channel') }} s, {{ source('sh\_database', 'countries') }} t, {{ source('sh\_database', 'customers') }} c WHERE s.cust\_id = c.cust\_id AND c.country\_id = t.country\_id dbt Core data source connections are defined using a configuration file in YAML format stored by default at $HOME/.dbt/profiles.yml, outside of the git repository files so database credentials don’t get published with your code by accident. dbt\_test: target: dev outputs: dev: type: oracle user: MARK\_DBT pass: \*\*\*\*\* protocol: "tcps" host: adb.uk-london-1.oraclecloud.com port: 1522 service: g71ab9f09757846\_radevelopmentadw\_high.adb.oraclecloud.com schema: MARK\_DBT retry\_count: 1 retry\_delay: 5 shardingkey: - skey supershardingkey: - sskey cclass: CONNECTIVITY\_CLASS purity: self threads: 1 There are various ways that dbt can connect to an Oracle database; using thin or thick drivers from the python-oracledb Oracle client libraries package along with the choice of either TLS (Transport Layer Security) or mutual TLS (mTLS) when connecting to Oracle Autonomous Data Warehouse or Autonomous Transaction Processing. I used the thin client mode and was able to connect to Oracle Autonomous Data Warehouse Cloud without using a wallet file, using just my database username, password and host/port/service name. dbt Core packages are run from the command-line and when executed, compile your SQL and Jinja model code into SQL SELECT and MERGE statements that transforms, tests and runs utilities to transform your data within the Oracle database. (dbt-oracle-venv) markrittman@marks-imac-4 dbt\_adbs\_test\_project % dbt build 21:47:47 Running with dbt=1.5.3 21:47:47 oracle adapter: Running in thin mode 21:47:47 Registered adapter: oracle=1.5.3 21:47:47 Found 17 models, 10 tests, 1 snapshot, 1 analysis, 637 macros, 3 operations, 1 seed file, 8 sources, 2 exposures, 0 metrics, 0 groups 21:47:47 21:47:50 21:47:50 Running 1 on-run-start hook 21:47:50 1 of 1 START hook: dbt\_adbs\_test\_project.on-run-start.0 ........................ \[RUN\] 21:47:50 1 of 1 OK hook: dbt\_adbs\_test\_project.on-run-start.0 ........................... \[OK in 0.03s\] 21:47:50 21:47:50 Concurrency: 1 threads (target='dev') 21:47:50 21:47:50 1 of 28 START sql table model MARK\_DBT.countries ............................... \[RUN\] 21:47:52 1 of 28 OK created sql table model MARK\_DBT.countries .......................... \[OK in 1.30s\] 21:47:52 2 of 28 START sql view model MARK\_DBT.direct\_sales\_channel\_promo\_cost .......... \[RUN\] 21:47:53 2 of 28 OK created sql view model MARK\_DBT.direct\_sales\_channel\_promo\_cost ..... \[OK in 0.88s\] 21:47:53 3 of 28 START sql table model MARK\_DBT.promotion\_costs ......................... \[RUN\] 21:47:54 3 of 28 OK created sql table model MARK\_DBT.promotion\_costs .................... \[OK in 1.30s\] 21:47:54 4 of 28 START sql table model MARK\_DBT.promotion\_costs\_for\_direct\_sales\_channel \[RUN\] 21:47:56 4 of 28 OK created sql table model MARK\_DBT.promotion\_costs\_for\_direct\_sales\_channel \[OK in 2.49s\] 21:47:56 5 of 28 START sql incremental model MARK\_DBT.promotion\_costs\_for\_direct\_sales\_channel\_incr\_insert \[RUN\] 21:47:59 5 of 28 OK created sql incremental model MARK\_DBT.promotion\_costs\_for\_direct\_sales\_channel\_incr\_insert \[OK in 2.37s\] 21:47:59 6 of 28 START sql incremental model MARK\_DBT.promotion\_costs\_for\_direct\_sales\_channel\_incr\_merge \[RUN\] 21:48:00 6 of 28 OK created sql incremental model MARK\_DBT.promotion\_costs\_for\_direct\_sales\_channel\_incr\_merge \[OK in 1.56s\] 21:48:00 7 of 28 START sql incremental model MARK\_DBT.promotion\_costs\_for\_direct\_sales\_channel\_incr\_merge\_unique\_keys \[RUN\] 21:48:02 7 of 28 OK created sql incremental model MARK\_DBT.promotion\_costs\_for\_direct\_sales\_channel\_incr\_merge\_unique\_keys \[OK in 1.78s\] 21:48:02 8 of 28 START sql table model MARK\_DBT.sales\_cost .............................. \[RUN\] 21:48:04 8 of 28 OK created sql table model MARK\_DBT.sales\_cost ......................... \[OK in 1.64s\] 21:48:04 9 of 28 START sql table model MARK\_DBT.sales\_internet\_channel .................. \[RUN\] 21:48:06 9 of 28 OK created sql table model MARK\_DBT.sales\_internet\_channel ............. \[OK in 2.20s\] 21:48:06 10 of 28 START sql table model MARK\_DBT.union\_customer\_sales ................... \[RUN\] 21:48:12 10 of 28 OK created sql table model MARK\_DBT.union\_customer\_sales .............. \[OK in 6.14s\] 21:48:12 11 of 28 START sql incremental model MARK\_DBT.us\_product\_sales\_channel\_ranking . \[RUN\] 21:48:14 11 of 28 OK created sql incremental model MARK\_DBT.us\_product\_sales\_channel\_ranking \[OK in 2.20s\] 21:48:14 12 of 28 START sql incremental model MARK\_DBT.us\_product\_sales\_channel\_ranking\_append \[RUN\] 21:48:17 12 of 28 OK created sql incremental model MARK\_DBT.us\_product\_sales\_channel\_ranking\_append \[OK in 2.62s\] 21:48:17 13 of 28 START seed file MARK\_DBT.seed ......................................... \[RUN\] 21:48:18 13 of 28 OK loaded seed file MARK\_DBT.seed ..................................... \[INSERT 5 in 0.96s\] 21:48:18 14 of 28 START test dbt\_constraints\_primary\_key\_countries\_country\_id ........... \[RUN\] 21:48:19 14 of 28 PASS dbt\_constraints\_primary\_key\_countries\_country\_id ................. \[PASS in 1.50s\] 21:48:19 15 of 28 START test not\_null\_countries\_country\_id .............................. \[RUN\] 21:48:21 15 of 28 PASS not\_null\_countries\_country\_id .................................... \[PASS in 1.14s\] 21:48:21 16 of 28 START test unique\_countries\_country\_id ................................ \[RUN\] 21:48:21 16 of 28 PASS unique\_countries\_country\_id ...................................... \[PASS in 0.62s\] 21:48:21 17 of 28 START sql view model MARK\_DBT.us\_seed\_customers ....................... \[RUN\] 21:48:22 17 of 28 OK created sql view model MARK\_DBT.us\_seed\_customers .................. \[OK in 0.90s\] 21:48:22 18 of 28 START snapshot MARK\_DBT.promotion\_costs\_snapshot ...................... \[RUN\] 21:48:24 18 of 28 OK snapshotted MARK\_DBT.promotion\_costs\_snapshot ...................... \[success in 2.28s\] 21:48:24 19 of 28 START sql table model MARK\_DBT.internet\_sales\_channel\_customers ....... \[RUN\] 21:48:29 19 of 28 OK created sql table model MARK\_DBT.internet\_sales\_channel\_customers .. \[OK in 4.41s\] 21:48:29 20 of 28 START sql table model MARK\_DBT.people ................................. \[RUN\] 21:48:32 20 of 28 OK created sql table model MARK\_DBT.people ............................ \[OK in 2.86s\] 21:48:32 21 of 28 START sql table model MARK\_DBT.eu\_direct\_sales\_channels\_promo\_costs ... \[RUN\] 21:48:34 21 of 28 OK created sql table model MARK\_DBT.eu\_direct\_sales\_channels\_promo\_costs \[OK in 1.92s\] 21:48:34 22 of 28 START test accepted\_values\_people\_gender\_\_Male\_\_Female ................ \[RUN\] 21:48:35 22 of 28 PASS accepted\_values\_people\_gender\_\_Male\_\_Female ...................... \[PASS in 1.13s\] 21:48:35 23 of 28 START test dbt\_constraints\_primary\_key\_people\_id ...................... \[RUN\] 21:48:36 23 of 28 PASS dbt\_constraints\_primary\_key\_people\_id ............................ \[PASS in 0.85s\] 21:48:36 24 of 28 START test not\_null\_people\_id ......................................... \[RUN\] 21:48:36 24 of 28 PASS not\_null\_people\_id ............................................... \[PASS in 0.74s\] 21:48:36 25 of 28 START test test\_count\_employees ....................................... \[RUN\] 21:48:37 25 of 28 PASS test\_count\_employees ............................................. \[PASS in 0.96s\] 21:48:37 26 of 28 START test unique\_people\_id ........................................... \[RUN\] 21:48:38 26 of 28 PASS unique\_people\_id ................................................. \[PASS in 1.05s\] 21:48:38 27 of 28 START test dbt\_constraints\_foreign\_key\_eu\_direct\_sales\_channels\_promo\_costs\_country\_id\_\_country\_id\_\_ref\_countries\_ \[RUN\] 21:48:39 27 of 28 PASS dbt\_constraints\_foreign\_key\_eu\_direct\_sales\_channels\_promo\_costs\_country\_id\_\_country\_id\_\_ref\_countries\_ \[PASS in 0.93s\] 21:48:39 28 of 28 START test relationships\_eu\_direct\_sales\_channels\_promo\_costs\_country\_id\_\_country\_id\_\_ref\_countries\_ \[RUN\] 21:48:40 28 of 28 PASS relationships\_eu\_direct\_sales\_channels\_promo\_costs\_country\_id\_\_country\_id\_\_ref\_countries\_ \[PASS in 0.75s\] 21:48:41 21:48:41 Running 2 on-run-end hooks 21:48:41 1 of 2 START hook: dbt\_adbs\_test\_project.on-run-end.0 .......................... \[RUN\] 21:48:41 1 of 2 OK hook: dbt\_adbs\_test\_project.on-run-end.0 ............................. \[OK in 0.03s\] 21:48:41 Running dbt Constraints 21:48:45 Creating primary key: COUNTRIES\_COUNTRY\_ID\_PK 21:48:49 Creating primary key: PEOPLE\_ID\_PK 21:48:53 Creating foreign key: FK\_2419558567 referencing countries \['country\_id'\] 21:48:54 Finished dbt Constraints 21:48:54 2 of 2 START hook: dbt\_constraints.on-run-end.0 ................................ \[RUN\] 21:48:54 2 of 2 OK hook: dbt\_constraints.on-run-end.0 ................................... \[OK in 0.00s\] 21:48:54 21:48:54 21:48:54 Finished running 9 table models, 2 view models, 5 incremental models, 1 seed, 10 tests, 1 snapshot, 3 hooks in 0 hours 1 minutes and 7.14 seconds (67.14s). 21:48:54 21:48:54 Completed successfully 21:48:54 21:48:54 Done. PASS=28 WARN=0 ERROR=0 SKIP=0 TOTAL=28 dbt packages typically include descriptions for fields, schema definitions and other metadata that, together with the DAG dependency graph defined by all of the ref() and source() definitions within your models can be served-up from the command line using a built-in webserver, like this: (dbt-oracle-venv) markrittman@marks-imac-4 dbt\_adbs\_test\_project % dbt docs generate 21:54:57 Running with dbt=1.5.3 21:54:57 oracle adapter: Running in thin mode 21:54:57 Registered adapter: oracle=1.5.3 21:54:57 Found 17 models, 10 tests, 1 snapshot, 1 analysis, 637 macros, 3 operations, 1 seed file, 8 sources, 2 exposures, 0 metrics, 0 groups 21:54:57 21:55:00 Concurrency: 1 threads (target='dev') 21:55:00 21:55:03 Running dbt Constraints 21:55:03 Finished dbt Constraints 21:55:04 Building catalog 21:56:12 Catalog written to /Users/markrittman/prod/dbt-oracle/dbt\_adbs\_test\_project/target/catalog.json (dbt-oracle-venv) markrittman@marks-imac-4 dbt\_adbs\_test\_project % dbt docs serve 22:06:41 Running with dbt=1.5.3 22:06:41 oracle adapter: Running in thin mode Serving docs at 8080 To access from your browser, navigate to: http://localhost:8080 Press Ctrl+C to exit. 127.0.0.1 - - \[06/Sep/2023 23:06:41\] "GET / HTTP/1.1" 200 - 127.0.0.1 - - \[06/Sep/2023 23:06:42\] "GET /manifest.json?cb=1694038002092 HTTP/1.1" 200 - 127.0.0.1 - - \[06/Sep/2023 23:06:42\] "GET /catalog.json?cb=1694038002092 HTTP/1.1" 200 - and then visualized as a data dictionary portal and transformation dependency graph, like this: ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872081826-s3qr4.jpg%3Fresize%3D1400%252C966%26ssl%3D1) #### Interested? Find Out More! Rittman Analytics is a boutique data analytics consultancy that’s both an [Oracle Partner](https://www.rittmananalytics.com/partners/oracle) with over 25 years experience working with Oracle technology and Oracle customers, and a [long standing partner](https://partners.getdbt.com/english/directory/partner/1540716/rittman-analytics) of [dbt Labs](https://www.rittmananalytics.com/partners/dbt-labs), the software company that sponsors dbt Core development. You can read more about our work with dbt [on our blog](https://blog.rittmananalytics.com/tagged/dbt), see examples of our dbt development approach in [our Github repo](https://github.com/rittmananalytics/ra_data_warehouse) and listen to episodes of our podcast, Drill to Detail, where [we discuss dbt and the analytics engineering approach](https://www.rittmananalytics.com/drilltodetail/2018/10/2/drill-to-detail-ep59-looker-dbt-and-digital-analytics-today-with-special-guests-tristan-handy-and-stewart-bryson) with one of dbt’s original authors, Tristan Handy. If you’re looking for some help and assistance adopting dbt and the analytics engineering approach to data warehouse development, or to help build-out your analytics capabilities and data team using a modern, flexible and modular data stack, [contact us now](https://calendly.com/markrittman/initial-discovery-call-with-mark-rittman?utm_source=medium.com&utm_medium=blog&utm_campaign=customer_segmentation) to organise a 100%-free, no-obligation call — we’d love to hear from you! [dbt](https://rittmananalytics.com/blog/tag/dbt)[Oracle](https://rittmananalytics.com/blog/tag/Oracle)[Oracle Cloud](https://rittmananalytics.com/blog/tag/Oracle+Cloud) [Mark Rittman](https://rittmananalytics.com/blog?author=5846da08f5e231d83261e221) CEO of Rittman Analytics, host of the Drill to Detail Podcast, ex-product manager and twice company founder. [https://rittmananalytics.com](https://rittmananalytics.com/) --- # ChatGPT-Powered Data Analysis using Cube, Delphi and the Code Interpreter Plugin Canonical: https://rittmananalytics.com/blog/2023/07/09/chatgpt-powered-data-analysis-using-cube-delphi-and-the-code-interpreter-plugin URL: https://rittmananalytics.com/blog/2023/07/09/chatgpt-powered-data-analysis-using-cube-delphi-and-the-code-interpreter-plugin Description: A couple of months ago I blogged about our experiments using ChatGPT to create dbt models and data transformations and I’m talking about this at the London dbt Meetup on… Author: Mark Rittman Published: 2023-07-09T09:00:00+01:00 Modified: 2026-07-02T00:36:00+01:00 --- ![ChatGPT-Powered Data Analysis using Cube, Delphi and the Code Interpreter Plugin](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F9be07-10xtmqjkx-wncvjq4ou43jg.png) A couple of months ago I blogged about [our experiments using ChatGPT to create dbt models and data transformations](https://www.rittmananalytics.com/blog/2023/3/26/chatgpt-large-language-models-and-the-future-of-dbt-and-analytics-consulting) and I’m talking about this at the [London dbt Meetup on July 12th 2023](https://www.meetup.com/london-dbt-meetup/events/294202277/), if you’re in the area that evening. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2Fbd17c-1emynqxhifl78ngldytb9zw.png) Since writing that blog back in May 2023, ChatGPT and ecosystem around OpenAI’s generative AI models has evolved at an impressive pace. As well as using this technology to [automate steps in our analytics engineering workflow](https://github.com/rittmananalytics/dwa#dwa-cube---using-the-large-language-model), we’ve also been looking at how it could be used to enhance or replace some of the work done by data analysts and data scientists. ChatGPT in its original, free-to-use version had some significant limitations if you wanted to use it for data analysis, the most significant of which was its inability to connect to the databases and other data sources that you might want to analyze. You could export data into CSV format, for example, and then use the paid [ChatGPT Plus](https://openai.com/blog/chatgpt-plus) service (with its option to not use your data for training its models) to put an excerpt of your data in one of your prompts, like this: ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F06110-1_ag4wyso3gzmdkb27uy_tg.png) and it would be able to parse and understand the contents. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F42478-1biq65bqecilhssibrsnltw.png) However if you ask ChatGPT to perform some analysis on this data it responds by saying that as a text-based AI model it can’t do that, and suggests some python code that I could run and get the answer with instead; a type of response that suited [my previous use-case of writing dbt code](https://www.rittmananalytics.com/blog/2023/3/26/chatgpt-large-language-models-and-the-future-of-dbt-and-analytics-consulting) well but isn’t all that helpful in this instance. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2Fca2d4-15bef_z6bjwnbnkjrjwj47q.png) What this basic version of ChatGPT can do though is summarise themes in the data I’ve given it, a task that text-based LLMs do well. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F9be07-10xtmqjkx-wncvjq4ou43jg.png) So how has ChatGPT’s data analytics capability moved on since then? * * * The most significant extension to ChatGPT’s capabilities that came along since my previous blog comes from its new [plugin store](https://openai.com/blog/chatgpt-plugins), available to anyone who is using their paid ChatGPT Plus version. By my quick count there must be over 400 plugins now available with a dozen or so to do with data analytics and others that can connect to Git repositories, spin-up notebooks or create data visualizations. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F016af-1eo1uivocqbdbscddo9wvng.png) One that seemed very relevant to our data analysis needs is the [Chat with your Data](https://chatwithyourdata.io/) plugin, letting you connect either a Google Sheets or Excel spreadsheet or a Postgres or Postgres-compatible database and query its contents using a conversational interface. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F1b454-1enh4feabs0e6i56v0dwlkg.png) Now our business [uses Google BigQuery for our cloud data warehouse](https://blog.rittmananalytics.com/how-rittman-analytics-does-analytics-part-2-building-a-modern-data-stack-using-dbt-google-a0a17cdb4812), not Postgres, but we’ve [recently been using Cube to create a semantic layer](https://cube.dev/events/how-rittman-analytics-delivers-the-semantic-layer-today) over our BigQuery warehouse and it comes with a Postgres-compatible SQL API that we could try and connect to this ChatGPT plugin. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F781d0-1xf9xs1bwitlridergos7eq.png) It needs a bit of guidance at the start to work out what tables contain the data you’re asking for; in the example below I asked it how many deals we’d recorded (Hubspot terminology for sales opportunities) and it couldn’t work out that the Sales view in Cube was where this measure was located. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2Fb82e1-1vudfcfwo2yg8lc0l1r0otg.png) The fact that this plugin had access to the data dictionary as well as the data was interesting and I’ll come back to that in a moment, but once everything was connected and the plugin knew its way around the data source, I was able to conversationally ask it to perform some basic analysis. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2Fcef91-1ggz0zenqjkl7mzaidh5-ra.png) In practice though I’d say it was a limited success; although the plugin can connect to Cube it did have a problem from time-to-time with the fact that Cube [generally serves-up data through its SQL API pre-aggregated](https://cube.dev/docs/backend/sql#querying-cube-tables-aggregated-vs-non-aggregated-queries), meaning that SUM() and other similar aggregation functions don’t work as the plugin expected and it also had problems with time range filters, but those are more problems unique to Cube as a data source rather than the API. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2Fbb110-1ilesmp-ce_ksyrqfdhdv1q.png) As I said earlier though this plugin provides access to the data dictionary for the data source as well its data, so we can pass this additional information to ChatGPT in a request to create dbt models for this data source that otherwise wouldn’t be in its training data. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F10565-1sku3zczrrvacmqxkgiueyq.png) As usual it gives you a basic rather than comprehensive answer, but the significance is in the potential it provides for ChatGPT to model and transform the client-specific application databases that clients typically want to you centralize, not just the ones with public schemas and Fivetran dbt models that ChatGPT is limited to understanding right now. * * * A more polished and professional version of this “conversational-AI-over-semantic-layers” style of data analysis was also the topic of one of our recent Drill to Detail podcast episodes, [Drill to Detail Ep.102 ‘LLMs, Semantic Models and Bringing AI to the Modern Data Stack’ with Special Guest David Jayatillake](https://www.rittmananalytics.com/drilltodetail/2023/4/14/ep102-llms-semantic-models-and-bringing-ai-to-the-modern-data-stack-with-special-guest-david-jayatillake) where we discussed his new startup and product, [Delphi](https://www.delphihq.com/). Delphi is a plug-in for Slack that connects to Looker, Cube, Lightdash and the dbt Semantic Layer and provides a conversational interface to your data powered by [OpenAI’s GPT-4, Anthropic’s Claude and open source AIs](https://delphihq.substack.com/p/delphi-is-multi-llm). ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2Fbc1b9-1ftqdxqvxzm-ytlyqu1lkbw.png) Taking this conversational-style chat UI one step further and mimicking the way a good analyst will check their assumptions about fields to use and results you’re looking for, Delphi checks those assumptions with you first before beginning its analysis with potentially a misunderstanding of what you’re looking for. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2Fee52c-1mwmws1yyys4ddpirr9tbgw.png) As would be the case with any data analyst working with your semantic layer for the first time, human or AI-powered, much of the usefulness and effectiveness of Delphi and the Chat with your Data plugin will rest on how well your semantic layer and/or database schema are structured and labelled. If you have identically or similarly-named measures and dimensions in multiple places or confusingly structured explores and cubes, Delphi will struggle to find what you’re looking for without your guidance. But like a good human data analyst it learns and if asked the same question again by you or someone else, it prompts you to use that analysis rather than start another analysis from scratch. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F41edc-1i0vndmen2ontp2bwjzmara.png) * * * Another approach to powering data analysis using ChatGPT is the [Code Interpreter plugin](https://openai.com/blog/chatgpt-plugins#code-interpreter) that recently became available for all ChatGPT Plus users. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F85810-17z8sgpjl7luupuitelo8ka.png) [Code Interpreter](https://openai.com/blog/chatgpt-plugins#code-interpreter) is an official ChatGPT plugin for data analytics, data science and coding that can write, and execute, python code. Unlike the coding example I went through in my previous blog where ChatGPT wrote code that I then had to copy and run in my own environment, the Code Interpreter plugin creates its own python virtual environment into which you can upload data and code files and download results. I start then by uploading a 229MB CSV file containing page view and other visitor interaction event data from our website, with my goal being to better understand what content is more likely to lead to a sales enquiry being made. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F62a24-16xnr-ynl9diivkghip7ngg.png) The Code Interpreter Plugin correctly describes the contents of my file, after which I give it a bit more context about what we consider as our conversion event and the particular entrance events I’m looking to analyze. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2Fb6b51-1y_fpybgzhi56womzc5ivgg.png) ChatGPT with the Code Interpreter plugin now has the ability to answer my original questions and if you click on the Show Work button in its response, you can see the python code that the Code Interpreter plugin used to answer your question. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F99ebd-1n-rhylrb3yhwivez7aj1w.png) The Code Interpreter plugin can also create data visualizations using the [matplotlib](https://matplotlib.org/) data visualization python library, and annotates the results with an interpretation of what they’re telling you. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F27907-1anejwzc3ip5x2tu_3ueiua.png) It’s actually pretty good and the back-and-forth conversation I had with it, asking questions and then following on with others as it presented back results to me were as good, I’d say, as most real-life conversations I’ve had with data scientists in the past. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F01ff8-1xrwmaqvb6rq5sgptvcdbyw.png) I then upload a second file containing [IP address block-to-geography lookups](https://github.com/datasets/geoip2-ipv4/blob/master/data/geoip2-ipv4.csv) and check that the Code Interpreter plugin knows what the data in the file is for. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F90e94-1t46ndlpzintn9schc-h3uq.png) Then I ask it to combine the geography lookup data with the events data it already has and give me a breakdown of conversions by country. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2Fb8fec-172thercamfehx571tbyy7w.png) Where the Code Interpreter Plugin did fall-apart, at least in my experience, was when I started asking it to do proper data science-type tasks such as creating a model to predict whether a user was likely to convert. In my instance it kept having to circle-back because columns it needed weren’t in its existing dataframe, then the number of feature values were too many for model type it wanted to use and then finally it ran out of memory and kept asking me to re-upload the two files, forgetting wher it had put them and asking me once more to re-upload them. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2Fd2c42-1ae2q3uphjcnconkw2x3dya.png) Even so, it was pretty impressive and the problems I hit are no doubt easy to fix either by my reading the documentation or OpenAI upping some memory limits for the Code Interpreter plugin. As of now all of these generative AI tools for data analysis, data engineering and analytics engineering all suffer from the [daleks-can’t-climb-stairs problem](https://www.theguardian.com/science/2005/mar/10/thisweekssciencequestions3#:~:text=As%20terrifying%20as%20Dr%20Who%27s,due%20to%20begin%20at%20Easter.) that a normal human analyst would work around instinctively but crucially, a well labelled and structured data model designed for use by business users will also be easily understood by a text-based LLM. So can ChatGPT and its various plugins and API ecosystem partners play the role of a data analyst or even replace one? Realistically its too early to say, but the conversational approach these plugins use is probably better suited to data analysis and business questions than data engineering and we already have clients that are looking to use products such as Delphi rather than hire new data analysts in the future. ### INTERESTED? FIND OUT MORE Rittman Analytics is a boutique analytics consultancy specializing in the modern data stack who can help you [centralise your data sources](https://markr320.sg-host.com/data-centralization), [optimize your marketing activity](https://markr320.sg-host.com/#:~:text=OUR%20SOLUTIONS-,Marketing%20Attribution,-Modern%20Data%20Stack) and [enable your end-users and data team](https://markr320.sg-host.com/data-team-enablement) with best practices and [modern analytics workflow](https://markr320.sg-host.com/getting-started-with-dbt) provided by a team of smart and AI-enhanced data and analytics engineers now looking relieved that its the data analysts who’ll be first to be replaced by an LLM. If you’re looking for some help and assistance analyzing and understanding your customer behaviour, or to help build-out your analytics capabilities and data team using a modern, flexible and modular data stack, [contact us now](https://calendly.com/markrittman/initial-discovery-call-with-mark-rittman?utm_source=medium.com&utm_medium=blog&utm_campaign=customer_segmentation) to organize a 100%-free, no-obligation call — we’d love to hear from you! --- # Wednesday Webinar Series : “How Rittman Analytics Builds Modern Data Stacks using Cube, Preset and Dagster” – Wednesday 28th June 2023 Canonical: https://rittmananalytics.com/blog/2023/06/19/wednesday-webinar-series-how-rittman-analytics-builds-modern-data-stacks-using-cube-preset-and-dagster-wednesday-28th-june-2023 URL: https://rittmananalytics.com/blog/2023/06/19/wednesday-webinar-series-how-rittman-analytics-builds-modern-data-stacks-using-cube-preset-and-dagster-wednesday-28th-june-2023 Description: Today we announced we’ll be deprecating non-last-click rules-based attribution models, which includes first-click, linear, positioned based & time decay, in Google Ads & GA4. > > Data-driven attribution (DDA) is recommended & last-click will remain available. > > More on why & timing: > > — AdsLiaison (@adsliaison) [April 6, 2023](https://twitter.com/adsliaison/status/1644080323292962816?ref_src=twsrc%5Etfw) [https://platform.twitter.com/widgets.js](https://platform.twitter.com/widgets.js) Data-driven attribution offers, in-theory, a number of benefits compared to “rules-based” models such as first-click and linear attribution, including being able to scale to very large datasets, theoretical reduction in bias and the ability to automatically adjust attribution weights as your marketing landscape evolves. And yet in-practice, having a marketing attribution model that changes the weighting it applies to email marketing interactions, for example, from year-to-year can make comparison reporting meaningless as you’ll never really know whether increased conversions attributed to that channel came from your increased spend or a change to Google’s model. More importantly, as a “black box” attribution model you’re reliant on trust rather than an ability to inspect rules and code when it comes to weightings applied to each marketing channel, and its important to remember that the primary purpose of Google Analytics 4 for Google is to drive more spend on Google Ads campaigns. However and as we covered in our blog [Event-Based Analytics (and BigQuery Export) comes to Google Analytics 4; How Does It Work… and What’s the Catch?](https://www.rittmananalytics.com/blog/2021/7/25/event-based-analytics-and-bigquery-export-comes-to-google-analytics-4-how-does-it-worknbsp-and-whats-thenbspcatch), Google Analytics 4 now allows you to export your event data in full-fidelity into Google BigQuery, opening-up the option to build your own first click, linear, time-decay and other rules-based marketing attribution models using logic you own and can change under your organisation’s control, not Google’s. As an example of how you can do this, we built our own rules-based attribution models for GA4 that provided first-click, linear, time-decay and last-click marketing attribution along with first paid click, last paid click and last non-direct click models, attributing conversions across multiple purchase cycles for each converter and displaying the results in the Looker dashboard shown below. The code for this blog can be downloaded in a couple of ways: - [A Github gist](https://gist.github.com/markrittman/b64c6dc04e39886c8b7870f357e7300f) containing a single Google BigQuery Standard SQL query that sources its event data from the [BigQuery GA Sample Dataset](https://developers.google.com/analytics/bigquery/web-ecommerce-demo-dataset) and provides an attribution dataset that can be used as a data source for Looker Studio, Preset, Metabase or any other BigQuery-compatible BI tool, outputting one row per converter conversion cycle per attribution model with columns for each of the measures (conversions, conversion value etc) - [A Looker block](https://github.com/rittmananalytics/ga4_marketing_attribution_looker_block) containing a single LookML view, model and LookML dashboard that uses the same underlying BigQuery SQL query but makes the attribution model output queryable interactively in a Looker explore, as shown below This particular set of attribution models was designed for B2C-type businesses where a customer first opens an account, requests a trial or performs a once-only activity that you want to attribute value for, which is then followed by one or more purchase conversions each of which needs to be attributed using its own “conversion” or “purchase” cycle. In the example code we use the `add_payment_info` GA4 event type as the once-only account opening conversion and the `purchase` event type as the repeating main conversion but these choices can be changed in the SQL code, and the purchase conversions we attribute the number and value of each conversion. The Looker dashboard allows you to choose which attribution model is used to attribute conversions and conversion value, along with a drop-down selector for the dimension by which those conversions are visualized; the Looker explore provided by the Looker block provides access to a wider set of breakdown dimensions based on default GA4 event properties. Finally, while we’ve provided the SQL and Looker code for this example of rules-based attribution using the BigQuery GA4 example data export dataset, you will of course need to adapt it to your own particular choice of attribution events, data export location and reporting needs.  The code is in a public Github repo and can be cloned, forked, reused or otherwise adapted for your own use, or we’d be happy to do that for you.  ### INTERESTED? FIND OUT MORE Rittman Analytics is a boutique data analytics consultancy and a Google Cloud Platform partner with many years experience working with BigQuery, GA and Looker delivering marketing attribution and other modern data stack implementations. If you’re looking for some help and assistance adapting this example code or to help build-out your analytics capabilities and data team using a modern, flexible and modular data stack, [contact us now](https://calendly.com/markrittman/initial-discovery-call-with-mark-rittman?utm_source=medium.com&utm_medium=blog&utm_campaign=customer_segmentation) to organize a 100%-free, no-obligation call — we’d love to hear from you! --- # Building a Mobile-Friendly KPI Dashboard using Looker Studio Integration with Looker Universal Semantic Model — Rittman Analytics Canonical: https://rittmananalytics.com/blog/2023/04/06/building-a-mobile-friendly-kpi-dashboard-using-looker-studio-integration-with-looker-universal-semantic-model-rittman-analytics URL: https://rittmananalytics.com/blog/2023/04/06/building-a-mobile-friendly-kpi-dashboard-using-looker-studio-integration-with-looker-universal-semantic-model-rittman-analytics Description: Building a Mobile-Friendly KPI Dashboard using Looker Studio Integration with Looker Universal Semantic Model Looker 6 Apr Written By Mark Rittman One of the new features added into Looker in… Author: Mark Rittman Published: 2023-04-06T01:00:00+01:00 Modified: 2023-04-06T01:00:00+01:00 --- ![Building a Mobile-Friendly KPI Dashboard using Looker Studio Integration with Looker Universal Semantic Model — Rittman Analytics](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2023%2F04%2F1752872134836-674xx.png%3Ffit%3D2202%252C1742%26ssl%3D1) # Building a Mobile-Friendly KPI Dashboard using Looker Studio Integration with Looker Universal Semantic Model [Looker](https://rittmananalytics.com/blog/category/Looker) 6 Apr Written By [Mark Rittman](https://rittmananalytics.com/blog?author=5846da08f5e231d83261e221) One of the new features added into Looker in the past year is [integration with Looker Studio](https://support.google.com/looker-studio/answer/12388266?hl=en#zippy=%2Cin-this-article), the [new name](https://searchengineland.com/google-data-studio-now-known-as-looker-studio-388658) for Google Data Studio, along with similar integration with Google Sheets. Users of Looker Studio can now build reports using governed metrics and dimensions from their [Looker semantic model](https://cloud.google.com/blog/products/data-analytics/lookers-universal-semantic-model) while Looker developers now have access to a much richer and flexible reporting tool should they want to go beyond standard Looker dashboards. Google Cloud Platform recently announced BI Connector integration for [ThoughtSpot](https://www.thoughtspot.com/press-releases/thoughtspot-announces-partnership-with-google-cloud-and-integrates-with-looker-to-transform-how-companies-use-data) and [Sisu](https://sisudata.com/blog/sisu-integration-for-looker) to go with the previously-announced [Tableau](https://www.tableau.com/blog/expanding-google-partnership-tableau-integrations-looker-bigquery), together with a new forthcoming tool called Looker Modeler that enables LookML development of what’s now termed the Looker Universal Semantic Model independent of the Looker web IDE. [https://platform.twitter.com/widgets.js](https://platform.twitter.com/widgets.js)","width":550,"height":null,"resolvedBy":"twitter","providerName":"Twitter"}” data-block-type=”22″ id=”block-yui\_3\_17\_2\_1\_1680737062333\_3574″\> > Looker roadmap from Google Data Cloud & AI Summit : 1/ Looker Modeler + LookML as a universal metrics layer w/Thoughtspot & Sisu as launch partners; 2/ Looker Google Cloud Core brings Looker to GCP Console; 3/ JDBC driver for SQL access to Looker models [https://t.co/SPe7maoLlX](https://t.co/SPe7maoLlX) > > — Mark Rittman (@markrittman) [April 1, 2023](https://twitter.com/markrittman/status/1642298147698253829?ref_src=twsrc%5Etfw) For now though, one way that we’ve used this new integration at Rittman Analytics is to create a mobile-friendly web app KPI dashboard to accompany our [main Looker KPI dashboard](https://www.rittmananalytics.com/blog/2023/2/18/kpi-dashboards-and-balanced-scorecards-using-looker-dbt-and-google-bigquery), showing in a compact and mobile-friendly format the most important business metrics for our management team and based-on the same “single version of the truth” as the main dashboard. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872134836-674xx.png%3Fresize%3D2202%252C1742%26ssl%3D1) * * * A pre-requisite for connecting Looker Studio to your Looker account is to first enable the **BI Connectors** feature within Looker’s Admin menu. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872134999-jl0nl.jpg%3Fresize%3D2048%252C1184%26ssl%3D1) Now, assuming that you’ve got an explore set-up within Looker that you want to use as your Looker Studio data source, use your web browser to navigate to the Looker Studio [Data Sources](https://lookerstudio.google.com/u/0/navigation/datasources) page. From there, click the **Create** button and select **Looker** as the data source type, then enter the domain name for your Looker account and finally the **Connect Looker Account** button to authorise and create your connection. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872135013-i6c0lc.jpg%3Fresize%3D1400%252C984%26ssl%3D1) You should now see a list of models that your Looker account makes available. Select the model, and then the explore within that model that contains the data you are looking for, and then press the Connect button. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872135079-zegco9.jpg%3Fresize%3D1400%252C934%26ssl%3D1) Note that, similar to the [Merge Query](https://cloud.google.com/looker/docs/merged-results) feature within Looker’s explore user interface, later on we can merge in additional Looker data sources into our report as well as data sourced from any of the other [official](https://support.google.com/looker-studio/answer/6268208?hl=en#zippy=%2Cin-this-article) and [community supported](https://developers.google.com/looker-studio/connector) data sources available for Looker Studio. If you’re familiar with the Data Studio report creation process you’ll recognise the page where all of the dimensions and measure columns from your data source are presented and available to hide, format or add more derived columns for. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872134994-2smkwn.jpg%3Fresize%3D1400%252C1051%26ssl%3D1) In the case of Looker-sourced measures you don’t need to set their aggregation method, just leave them as **Number** as the LookML definition of those measures’ aggregation type will automatically be used when displayed in a report; just press **Create Report** to start creating your report layout. Creating a Looker Studio report with Looker as the data source adds a folder-style grouping of the views, dimensions and measures in your chosen explore, and you can even click on the information icon to see the LookML definition for each of the explore items. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872135012-woqbth.jpg%3Fresize%3D1400%252C1090%26ssl%3D1) To make my Looker Studio report work as a web app-style mobile dashboard I then make the following configuration changes to a standard report template: 1. Select **Page** \> **Current page settings** 2. Set the **Canvas Size** to 390 (px) **Width** and 802 (px) **Height**, the typical viewport screen area for a modern iPhone. 3. Now press the **Theme and Layout** button in the report header menu and select **Auto hide** as the **Header visibility** setting. 4. Set the **Display Mode** to **Fit to width** 5. Finally, uncheck the **Has margin** checkbox ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872135027-jo5u69.jpg%3Fresize%3D1030%252C1400%26ssl%3D1) Now the report template is all set to be mobile-friendly and formatted correctly, I then create my mobile KPI dashboard using dark mode colours, sparkline charts and stripped-back bar charts and KPI tiles that call-out our most important performance metrics. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872134991-5pmeq5.jpg%3Fresize%3D1400%252C1056%26ssl%3D1) Notice the three additional data sources that were added to the final report to create the final mobile KPI dashboard shown in the screenshot above; two more Looker explores that contained the **Net Margin** and **Forecast Billing** KPIs, and the [Google Search Console](https://developers.google.com/search/blog/2022/03/connecting-data-studio) data source for the Search Traffic KPI. Using Looker Studio data source connectors to blend-in data from marketing, advertising and SaaS data sources is a useful lightweight alternative to using tools such as Fivetran and dbt to extract and then integrate those data sources into your warehouse. The downside is that any integration of those data sources is specific to just that one report, but if all you want to do is display your ad spend data alongside other marketing metrics then this approach gets the job done without requiring IT support and the more complete integration [I blogged-about a few posts ago](https://www.rittmananalytics.com/blog/2022/12/02/improving-wordpress-search-keyword-performance-using-looker-google-search-console-and-fivetran). ## **INTERESTED? FIND OUT MORE** Rittman Analytics is a boutique analytics consultancy specializing in the modern data stack who can help you [centralise your data sources](https://markr320.sg-host.com/data-centralization), [optimize your marketing activity](https://markr320.sg-host.com/#:~:text=OUR%20SOLUTIONS-,Marketing%20Attribution,-Modern%20Data%20Stack) and [enable your end-users and data team](https://markr320.sg-host.com/data-team-enablement) with best practices and a [modern analytics workflow](https://markr320.sg-host.com/getting-started-with-dbt). If you’re looking for some help and assistance analyzing and understanding your customer behaviour, or to help build-out your analytics capabilities and data team using a modern, flexible and modular data stack, [contact us now](https://calendly.com/markrittman/initial-discovery-call-with-mark-rittman?utm_source=medium.com&utm_medium=blog&utm_campaign=customer_segmentation) to organize a 100%-free, no-obligation call — we’d love to hear from you! [Looker](https://rittmananalytics.com/blog/tag/Looker) [Mark Rittman](https://rittmananalytics.com/blog?author=5846da08f5e231d83261e221) CEO of Rittman Analytics, host of the Drill to Detail Podcast, ex-product manager and twice company founder. [https://rittmananalytics.com](https://rittmananalytics.com/) --- # ChatGPT, Large Language Models and the Future of dbt and Analytics Consulting — Rittman Analytics Canonical: https://rittmananalytics.com/blog/2023/03/26/chatgpt-large-language-models-and-the-future-of-dbt-and-analytics-consulting-rittman-analytics URL: https://rittmananalytics.com/blog/2023/03/26/chatgpt-large-language-models-and-the-future-of-dbt-and-analytics-consulting-rittman-analytics Description: ChatGPT, Large Language Models and the Future of dbt and Analytics Consulting Artificial Intelligence 26 Mar Written By Mark Rittman Everyone’s talking about ChatGPT, generative AI models and the impact… Author: Mark Rittman Published: 2023-03-26T01:00:00+01:00 Modified: 2023-03-26T01:00:00+01:00 --- ![ChatGPT, Large Language Models and the Future of dbt and Analytics Consulting — Rittman Analytics](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2023%2F03%2F1752872155172-90f8wj.png%3Ffit%3D2338%252C1388%26ssl%3D1) # ChatGPT, Large Language Models and the Future of dbt and Analytics Consulting [Artificial Intelligence](https://rittmananalytics.com/blog/category/Artificial+Intelligence) 26 Mar Written By [Mark Rittman](https://rittmananalytics.com/blog?author=5846da08f5e231d83261e221) Everyone’s talking about ChatGPT, generative AI models and the impact they are likely to have on [white collar jobs](https://www.theatlantic.com/ideas/archive/2023/01/chatgpt-ai-economy-automation-jobs/672767/), software engineering (e.g. [Github Co-Pilot](https://github.com/features/copilot)) and more closer-to-home, [data roles within the modern data stack](https://roundup.getdbt.com/p/ep-40-what-can-generative-ai-do-for). We can even use generative AI to create an opening illustration for this blog, reassuring illustrators at least that their jobs are relatively safe for the time-being. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872155172-90f8wj.png%3Fresize%3D2338%252C1388%26ssl%3D1) [Rittman Analytics](https://www.rittmananalytics.com/about/who-we-are) is a typical “boutique” modern data stack consultancy with 8 team members including myself, all of us hands-on practitioners working with data and clients every day and providing a range of services from strategy advice through to implementation and user enablement. We typically have around half-a-dozen client projects running concurrently at any one time, spend quite a lot of time in face-to-face (usually virtual, these days) meetings with clients, project manage things on our side and take responsibility for making sure the overall project gets delivered as expected. We’re proud of our skills and testimonials from our clients … but so were milkmen, bus conductors, high-street travel agents, [leech collectors, toad doctors and gong farmers](https://stacker.com/money/50-jobs-no-longer-exist) just before their jobs became obsolete. So can ChatGPT-4, the latest version of OpenAI’s Large Language Model interactive chat service, replace the role of an analytics engineer or even the need for an analytics consultancy? * * * I start a new ChatGPT session on the [OpenAI website](https://chat.openai.com/auth/login) and give it a starting brief for an analytics project for a consulting business. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872155116-e1xen.png%3Fresize%3D1778%252C2036%26ssl%3D1) ChatGPT goes straight into defining a reasonable starting schema, though it makes a basic SQL syntactical error right at the get-go. BigQuery does, in-fact, now support primary and foreign key constraints but doesn’t enforce them, using them instead to provide additional schema metadata to BigQuery’s query planner and the correct syntax for this new DDL feature is PRIMARY KEY (column\_name) NOT ENFORCED. Another more subtle but, longer-term more significant, shortfall in ChatGPT’s approach is that it only models the bare minimum table schema that would satisfy the literal request from the user, and not a more comprehensive schema that either experience, or further questioning, would lead you to create. But excepting the syntax error in each table’s DDL, it does create a reasonable schema for the request it’s been given. I then ask ChatGPT to create a dbt package that takes the raw data from each of the Fivetran-sourced staging datasets, integrates and transforms it and then loads it into the fact and dimension tables it just defined a schema for. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872155054-huugaa.png%3Fresize%3D1542%252C2050%26ssl%3D1) I then give ChatGPT a more challenging task, to deduplicate client names not only when they are an exact match, but also when they just sound similar. [This often turns out to be a requirement](https://www.rittmananalytics.com/blog/2021/1/17/deduplicating-dbt-saas-data-warehousing) when working with company names from a source such as Hubspot where reps input many variations on a single company’s name when recording new sales deals, but you want those deals to resolve to a single company name when viewing that account’s deal history. ChatGPT confidently offers to do this using BigQuery’s Jaro-Winkler string similarity function, and rewrites the dbt code for me. Neat — I hadn’t even heard of this function. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872155226-kqbc8.png%3Fresize%3D1542%252C2588%26ssl%3D1) Except … BigQuery doesn’t have a JARO\_WINKLER string function. The [nearest you can get](https://stackoverflow.com/questions/39885358/google-bigquery-udf-for-jaro-winkle-distance) at the time of writing this blog is to create a UDF (User-Defined Function) and the only database server that I’m aware of that has Jaro-Winkler functionality is Oracle Database and even then, it’s a [parameter given to Oracle’s UTL\_MATCH function](https://oracle-base.com/articles/11g/utl_match-string-matching-in-oracle#jaro_winkler) rather than a SQL function in its own right. ChatGPT in-fact seems to have invented its own BigQuery SQL function and then confidently given it to me as its solution, but this code just won’t run and anyone handed it without BigQuery development experience just wouldn’t have a clue what is wrong and what to do next. I go on then to make further requests, and to be fair ChatGPT handles them well with a clear knowledge of what set of columns each source provides and how they map into common concepts such as invoices, invoice IDs and invoice dates as well as concepts such as paid, draft and outstanding invoices. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872155216-2oavl.png%3Fresize%3D1778%252C1868%26ssl%3D1) It also doesn’t flinch when I ask it to add calculations for employee utilisation, though in-practice we’d typically add a date spine table into the package to help calculate total capacity over all days, not just those on which an employee had actually worked. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872155208-k5kxzn.png%3Fresize%3D1790%252C1898%26ssl%3D1) I even ask ChatGPT to bring-in data from our HRMS system, [Humaans](https://humaans.io/), suggesting that it uses that service’s REST API to create a data extractor for data that it then loads into our BigQuery database. It then provides an overview of its solution and generates the Python code to connect to Humaan’s REST API, download the relevant data and then load it into BigQuery using its Python client. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872155328-ou989t.png%3Fresize%3D1778%252C1868%26ssl%3D1) It’s not the most robust solution and didn’t consider how the Python scripts or indeed the dbt package would be scheduled and orchestrated, but it’s a good initial prototype. More significantly though if you’re a freelancer taking on dbt work [on sites such as Fiverr](https://www.fiverr.com/sourav_sikka/create-data-pipelines-using-dbt-ie-data-build-tool) or a junior developer learning your trade by churning out dbt code tightly-specified by a solution architect, ChatGPT can most probably do that job better. It can even incorporate our naming standards in the code it generates. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872155186-5jfplr.png%3Fresize%3D1778%252C1868%26ssl%3D1) For now, although ChatGPT retains context from earlier instructions within the same overall chat session we’d need to provide details of these standards in any new chat session we initiated. In the (near) future though it will be possible for consultancies and other organisations to create their own LLMs (Large Language Models) that [incorporate domain-specific training data](https://www.ycombinator.com/launches/IEB-gloo-empower-llm-s-with-domain-specific-knowledge) such as our corpus of dbt code sitting in private git repos along with our naming standards, standard operating procedures and solution playbooks that together form the value and IP behind our service offerings. Some things, such as more recent features added to dbt, will trip up a GPT model trained on data collected only up to a certain point in-time. When I ask it to add a [dbt Semantic Layer](https://blog.rittmananalytics.com/the-dbt-semantic-layer-data-orchestration-and-the-modern-enterprise-data-stack-78d9d9ed5c18) over the models already created in the package it responds initially with an answer based on exposures, a pre-cursor to metrics and the semantic layer in dbt and then proposes a SQL view to provide a denormalized set of columns for that exposure. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872155177-k09pjg.png%3Fresize%3D1778%252C1200%26ssl%3D1) I then correct it and suggest a solution that uses the metrics feature introduced with more recent versions of dbt, to which ChatGPT responds first with an apology and then a revised solution that’s not a bad start to incorporating metrics and a semantic layer into the package. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872155280-isoc2s.png%3Fresize%3D2048%252C1384%26ssl%3D1) Where this does fall short in-terms of a response you’d expect from a [dbt Preferred Partner such as ourselves](https://partners.getdbt.com/english/directory/partner/1540716/rittman-analytics), though, is that there’s much more to consider here than just the syntax of the commands to implement this particular feature. By using dbt’s Semantic Layer you’re implicitly making the choice to limit yourself to those BI and other tools within the modern data stack ecosystem that are compatible with dbt’s Semantic Layer, Proxy Server and other enabling technologies; similarly, if you instead choose alternatives such as Looker’s Universal Semantic Model or Cube’s currently more feature-rich, but also more niche, semantic model then you’re implicitly making choices about the wider solution that won’t be apparent to a client who knows little about this area of technology. ChatGPT isn’t short of an opinion on technology choices though, if you ask it to give you a technology recommendation. After initially asking it to give me a way to sync data from the warehouse back into our Hubspot CRM, I then ask it whether a [Reverse ETL tool](https://www.rittmananalytics.com/blog/2021/2/15/customer-360-degree-analysis-and-hightouch) would be a better solution and if so, to recommend one. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872155140-30n86.png%3Fresize%3D1542%252C2682%26ssl%3D1) Nicely for Hightouch it recommends their tool, which reminds me to try and seed ChatGPT’s training data with references to Rittman Analytics so that it recommends us for dbt consulting services, if only for enquiries for those services where the consultancy is UK based, in Brighton, is a boutique consultancy and has a strong social media presence. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872155257-14k62.png%3Fresize%3D1508%252C3170%26ssl%3D1) The Data Shed, whoever the hell they are, are based in Leeds and seem to focus on Azure and AWS-based modern data stack solutions rather than ones based on dbt, Looker and BigQuery (hello guys if you’re reading this and wondering how you got yourselves dragged into this one, no offence implied). ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872155326-347jzo.png%3Fresize%3D1400%252C1021%26ssl%3D1) Dataform are in-fact a now-defunct product startup building a competitor to dbt that has recently been incorporated into Google Cloud Platform, so no idea why they were the next recommendation for these services. Most bizarrely, Count doesn’t even exist except as the tag-line used by a local book-keeping service, and its not until I prompt ChatGPT with our actual company name that it finally endorses us; I can see LLM optimization joining SEO optimization as an essential part of our future online marketing strategy. * * * So fittingly, I ask ChatGPT how it could best be used to improve our consultancy’s products, services and internal processes. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872155253-43q6m.png%3Fresize%3D1630%252C2162%26ssl%3D1) And it certainly handles the relatively simple, or at least relatively easy to define, tasks that are put to it, and doesn’t even try to conveniently forget to to the housekeeping tasks such as writing package documentation that takes time away from coding and other more interesting work. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872155168-cod8ac.png%3Fresize%3D1631%252C2118%26ssl%3D1) But it makes mistakes, and howlers such as the JARO\_WINKLER one in this project, and only does what you ask it to do and no more. As such it’s essentially another junior developer, full of enthusiasm, opinions and [plausible bullshit](https://www.transformingsociety.co.uk/2023/02/10/chatgpt-the-worlds-largest-bullshit-machine/) that if supervised and instructed properly can multiply the productivity of a senior consultant but you wouldn’t dare leave alone with a client to advise on their data analytics strategy. It’s even more of a dangerous prospect in the hands of an otherwise prospective client who knows what they want, thinks they know how to do it and just needs to find someone who can just do what he or she asks. These are often the most dangerous (but also entertaining and open-minded) clients who more than anything need saving from themselves and steering into the solution that actually meets their needs rather than delivering exactly what they asked for and it blowing-up shortly thereafter. Make no mistake — future iterations of generative AI, public services such as ChatGPT and domain-specific versions on these underlying models will make IT and computing to-date look like the spinning jennys that were the start of the industrial revolution. But there’s more to delivering a successful, valuable and scalable modern data stack implementation than just writing dbt, Looker and Python code, starting with defining the actual problem to be solved and ensuring that what gets delivered makes sense, solves the client’s problem and enables the types of business transformation and innovative services described by businesses such as [Breakthrough](https://www.rittmananalytics.com/case-studies/breakthrough), [Conversion Interactive Agency](https://www.rittmananalytics.com/case-studies/conversionia) and [Torticity](https://www.rittmananalytics.com/case-studies/torticity) in new client testimonials just added to our website. ## **INTERESTED? FIND OUT MORE** Rittman Analytics is a boutique analytics consultancy specializing in the modern data stack who can help you [centralise your data sources](https://markr320.sg-host.com/data-centralization), [optimize your marketing activity](https://markr320.sg-host.com/#:~:text=OUR%20SOLUTIONS-,Marketing%20Attribution,-Modern%20Data%20Stack) and [enable your end-users and data team](https://markr320.sg-host.com/data-team-enablement) with best practices and [modern analytics workflow](https://markr320.sg-host.com/getting-started-with-dbt) provided by a team of smart and experienced consultants managing a team of robotic, enthusiastic bullshitting large language models that do all the grunt work. If you’re looking for some help and assistance analyzing and understanding your customer behaviour, or to help build-out your analytics capabilities and data team using a modern, flexible and modular data stack, [contact us now](https://calendly.com/markrittman/initial-discovery-call-with-mark-rittman?utm_source=medium.com&utm_medium=blog&utm_campaign=customer_segmentation) to organize a 100%-free, no-obligation call — we’d love to hear from you! [ChatGPT](https://rittmananalytics.com/blog/tag/ChatGPT)[LLMs](https://rittmananalytics.com/blog/tag/LLMs)[Analytics Projects](https://rittmananalytics.com/blog/tag/Analytics+Projects) [Mark Rittman](https://rittmananalytics.com/blog?author=5846da08f5e231d83261e221) CEO of Rittman Analytics, host of the Drill to Detail Podcast, ex-product manager and twice company founder. [https://rittmananalytics.com](https://rittmananalytics.com/) --- # Webinar @ Mar 22 at 9am PT / 5pm GMT : “How Rittman Analytics delivers the semantic layer today with Cube” Canonical: https://rittmananalytics.com/blog/2023/03/16/webinar-mar-22-at-9am-pt-5pm-gmt-how-rittman-analytics-delivers-the-semantic-layer-today-with-cube URL: https://rittmananalytics.com/blog/2023/03/16/webinar-mar-22-at-9am-pt-5pm-gmt-how-rittman-analytics-delivers-the-semantic-layer-today-with-cube Description: In the upcoming webinar on Mar 22, 2023 with our friends at Cube , find out how we help data-driven companies spread the use of high-quality and consistent data throughout… Author: Mark Rittman Published: 2023-03-16T20:08:17+01:00 Modified: 2023-03-16T20:08:17+01:00 --- In the upcoming webinar on Mar 22, 2023 with our friends at [Cube](https://cube.dev/events/how-rittman-analytics-delivers-the-semantic-layer-today?utm_campaign=rittman-analytics-webinar&utm_medium=purple&utm_source=cube.dev) , find out how we help data-driven companies spread the use of high-quality and consistent data throughout their organisations using semantic layers. The webinar will feature [Olivier Dupuis](https://www.linkedin.com/in/ACoAAAEzYnUBbay3EUrSPLtp5wQchezZkorg-Kw), Chief Product Officer at Rittman Analytics along with [Lewis Baker](https://www.linkedin.com/in/ACoAABjWtToBv_74pulomJXueYAvl0v19jY4SnM), Chief Operations Officer at Rittman Analytics, joined by [Brian Bickell](https://www.linkedin.com/in/ACoAAAQ_sGkBwje-ZpWHWLSrxWzLCJQHDaE8BUQ) and [Igor Lukanin](https://www.linkedin.com/in/ACoAAA0_yW0BjsWGMc8isyipn1AszzDE00d2tMI) from Cube. Olivier and Lewis will demonstrate how we at Rittman Analytics take on a pragmatic approach to building semantic layers using mature and proven tools from the Modern Data Stack. You’ll follow an almost instantaneous journey from zero to a working semantic layer delivering data to end users and learn how to maintain dry yet robust data workflows with open-source tools like [Droughty](https://pypi.org/project/droughty/), [Dagster](https://dagster.io/), and [Cube](https://cube.dev/events/how-rittman-analytics-delivers-the-semantic-layer-today?utm_campaign=rittman-analytics-webinar&utm_medium=purple&utm_source=cube.dev). As a part of the experience, you’ll also explore what a feature-complete semantic layer is capable of. Our friends at Cube will demonstrate how their product provides consistency, visualisation agnosticity, and query acceleration that enables your end users to work with data in Tableau, Hex, React, or a variety of other tools while enjoying sub-second latencies. Make sure you [sign up today](https://cube.dev/events/how-rittman-analytics-delivers-the-semantic-layer-today) and put this event in your calendar, only 6 days left! --- # Webinar @ Mar 22 at 9am PT / 5pm GMT : “How Rittman Analytics delivers the semantic layer today with Cube” Canonical: https://rittmananalytics.com/blog/2023/03/16/2023-2-20-cube-semantic-layer-webinar URL: https://rittmananalytics.com/blog/2023/03/16/2023-2-20-cube-semantic-layer-webinar Description: In the upcoming webinar on Mar 22, 2023 with our friends at Cube , find out how we help data-driven companies spread the use of high-quality and consistent data throughout… Author: Mark Rittman Published: 2023-03-16T19:08:17+00:00 Modified: 2023-03-16T19:08:17+00:00 --- In the upcoming webinar on Mar 22, 2023 with our friends at [Cube](https://cube.dev/events/how-rittman-analytics-delivers-the-semantic-layer-today?utm_campaign=rittman-analytics-webinar&utm_medium=purple&utm_source=cube.dev) , find out how we help data-driven companies spread the use of high-quality and consistent data throughout their organisations using semantic layers. The webinar will feature [Olivier Dupuis](https://www.linkedin.com/in/ACoAAAEzYnUBbay3EUrSPLtp5wQchezZkorg-Kw), Chief Product Officer at Rittman Analytics along with [Lewis Baker](https://www.linkedin.com/in/ACoAABjWtToBv_74pulomJXueYAvl0v19jY4SnM), Chief Operations Officer at Rittman Analytics, joined by [Brian Bickell](https://www.linkedin.com/in/ACoAAAQ_sGkBwje-ZpWHWLSrxWzLCJQHDaE8BUQ) and [Igor Lukanin](https://www.linkedin.com/in/ACoAAA0_yW0BjsWGMc8isyipn1AszzDE00d2tMI) from Cube. Olivier and Lewis will demonstrate how we at Rittman Analytics take on a pragmatic approach to building semantic layers using mature and proven tools from the Modern Data Stack. You’ll follow an almost instantaneous journey from zero to a working semantic layer delivering data to end users and learn how to maintain dry yet robust data workflows with open-source tools like [Droughty](https://pypi.org/project/droughty/), [Dagster](https://dagster.io/), and [Cube](https://cube.dev/events/how-rittman-analytics-delivers-the-semantic-layer-today?utm_campaign=rittman-analytics-webinar&utm_medium=purple&utm_source=cube.dev). As a part of the experience, you’ll also explore what a feature-complete semantic layer is capable of. Our friends at Cube will demonstrate how their product provides consistency, visualisation agnosticity, and query acceleration that enables your end users to work with data in Tableau, Hex, React, or a variety of other tools while enjoying sub-second latencies. Make sure you [sign up today](https://cube.dev/events/how-rittman-analytics-delivers-the-semantic-layer-today) and put this event in your calendar, only 6 days left! --- # Building up a semantic layer with dbt Metrics, Cube and Droughty Canonical: https://rittmananalytics.com/blog/2023/01/13/2023-2-24-building-up-a-semantic-layer-with-dbt-metrics-cube-and-droughty URL: https://rittmananalytics.com/blog/2023/01/13/2023-2-24-building-up-a-semantic-layer-with-dbt-metrics-cube-and-droughty Description: I want to live in a world where data platform technologies can coexist together. And if they won’t do it willingly, I’m happy to force their hand a little That’s… Author: Mark Rittman Published: 2023-01-13T10:43:00+00:00 Modified: 2023-01-13T10:43:00+00:00 --- I want to live in a world where data platform technologies can coexist together. And if they won’t do it willingly, I’m happy to force their hand a little That’s the situation I’m in when it comes to the semantic layer where I want dbt Labs and Cube to just be the BFF they’re meant to be. The analytics engineering world has been abuzz with the semantic layer in the past year. Beyond the hype, what is its purpose and how do you implement it? Let’s go through a real-world example using the [discursus project](https://www.discursus.io/) as a playground. Gartner defines semantic data models as: > *A method of organizing data that reflects the basic meaning of data items and the relationships among them. > (source: [Gartner glossary](https://www.gartner.com/en/information-technology/glossary/semantic-data-model))* The components of a semantic layer are: - Domain-specific entities, such as *users* and *transactions* - Relationships between those entities - Entity attributes, that can loosely be classified as either keys, dimensions, and measures/metrics - Descriptions of the entities and their attributes **With those components in place, an end user is able to interact with entities and formulate domain-specific questions that return meaningful answers.** Let’s work this all out through an example.​ ## **Architecture** Building up a semantic layer requires that we start architecting those components into our data platforms and having a mechanism to surface them to end users. Going back to the discursus project, we have 3 key components in our architecture that play a role in serving the semantic layer: The discursus architecture and its components supporting the semantic layer - **[dbt](https://www.getdbt.com/)**: to define entities, attributes, relationships and metrics - **[Droughty](https://pypi.org/project/droughty/)**: to generate semantic definitions - **[Cube](https://cube.dev/)**: to expose the semantic layer’s components ## **Defining your semantic layer** ​At the heart of our semantic layer is dbt. This is where we transform our raw data into a semantic structure that is a reflection of our domain: entities, attributes and relationships. The discursus domain The discursus entities, attributes and relationships dbt is recognizing its role as the driving force in defining data’s semantic meaning. That meaning becomes in itself the API for other tools to interact with the data stored in a warehouse. dbt started unveiling its semantic strategy with the [metric concept](https://docs.getdbt.com/docs/build/metrics). As this is usually the main thing we interact with downstream (e.g. revenue, active users, inventory, etc.), it makes sense for dbt to provide a syntax to define those. A very simple example using the discursus domain is the following: ``` version: 2 metrics: - name: event_count label: Events count model: ref('events_fct') description: "The count of events" calculation_method: count expression: event_pk timestamp: event_date time_grains: [day, week, month, quarter, year] dimensions: - movement_fk ``` The above example simply tells downstream apps that our data domain includes a metric named *event\_count* which is *the count of events* by *daily*, *weekly*, etc. time grains and which can be sliced by the *movement\_fk* dimension. That definition sits on top of a model named *events\_fct*. And so our DAG now includes a new metric node. discursus DAG which includes a metric node Notice that I said that the metric is defined on top of a model. But really, we should be saying that a metric is an attribute of an entity. In our example above, we would then be saying that *event\_count* is the number of occurrences of the *event* entity. And that is exactly what dbt Labs is working on as their next step of deploying the semantic layer. There is an open conversation in the *dbt-core* git repository about [introducing the entity concept](https://github.com/dbt-labs/dbt-core/issues/6379) in their syntax. How that will manifest is beyond the scope of this post, but it’s a very interesting read for anyone curious about how that syntax is evolving. ## **Serving your semantic layer** Although dbt is making interesting contributions to the semantic layer, and there is a [growing ecosystem of integrations](https://www.getdbt.com/product/semantic-layer-integrations), the semantic layer is fragmented. Different syntaxes and engines are available, but none has won the day yet. It’s not uncommon for a data platform to have a semantic layer with the following components: - **dbt**: to define entities, attributes, relationships and field descriptions - **Looker (LookML)**: to provide self-serve BI capabilities for internal users - **Cube**: to serve data to external-facing apps - **dbml**: to document and share the latest state of a platform’s data assets Each of those components has its own syntax. That is a lot of manual work to port your semantic definitions from one tool to another… and it’s prone to errors. **That’s where you need a tool like Droughty to build bridges between your semantic layer components.** Reading from the data warehouse’s schemas, Droughty auto-generates semantic definition files. For example, I can simply run this: ``` droughty cube - profile-dir ./droughty_profiles.yml - project-dir ./droughty_projects.yml ``` And it will generate the following Cube definitions: ``` cube(`events_fct`, { sql: `select * from analytics.events_fct`, joins: { observations_fct: { relationship: `hasMany`, sql: `${CUBE.event_pk} = ${observations_fct.event_fk}`, }, }, dimensions: { action_geo_country_code: { sql: `action_geo_country_code`, type: `string`, description: `action geo country codes (agcc) are codes assigned to countries of the world by the action geo data company. the codes consist of five numbers and are used to classify countries by geographical region. action geo country codes are used in data analysis related to geospatial applications, including mapping, marketing, and population studies.`, }, event_date: { sql: `event_date`, type: `time`, description: `event date is the day or days that a particular event is scheduled to take place.`, }, event_pk: { primaryKey: true, type: `string`, sql: `event_pk`, description: `an event primary key is a unique identifier associated with an event, such as a conference, corporate gathering, or other special event. the primary key is typically assigned to each attendee and is used to help control event access and track attendance.`, }, } }); cube(`events_fct_extended`, { sql: `select * from ANALYTICS_QA.events_fct`, extends: events_fct, measures: { count_of_event_pk: { sql: `event_pk`, type: `count`, }, } }); ``` Auto-generating semantic definition files accelerate the flow from modifying your semantic layer in dbt and exposing those changes to your semantic engines. Below is an example of how I can now query my data warehouse with Cube using the semantic elements that were defined in dbt. ## **Conclusion** As [stated in a prior post](https://medium.com/@odupuis/dbt-inside-setting-the-data-platform-standard-419aa3d9e11c), dbt is positioning itself to be the data platform standard. The dbt semantic syntax is promising and is worth integrating into data platforms. But that can’t in itself drive an entire data platform’s semantic layer. We rely on other tools, which are well-established, mature and reliable. That includes Cube which is a great API solution for your data. It has its own syntax to define the semantic meaning of your data, but it shouldn’t be a pain to derive its definitions from what has already been defined through dbt. Tools like Droughty are necessary when you want to build a data platform that can use what’s best from different ecosystems. Without formal integrations between certain tools, it’s up to us data practitioners to force their hands a little. --- # Customer First Order Segmentation using Looker and Google BigQuery — Rittman Analytics Canonical: https://rittmananalytics.com/blog/2022/12/21/customer-first-order-segmentation-using-looker-and-google-bigquery-rittman-analytics URL: https://rittmananalytics.com/blog/2022/12/21/customer-first-order-segmentation-using-looker-and-google-bigquery-rittman-analytics Description: Customer First Order Segmentation using Looker and Google BigQuery LookerData Analytics 21 Dec Written By Mark Rittman Along with Recency, Frequency and Monetary Value (RFM) segmentation, one of the most… Author: Mark Rittman Published: 2022-12-21T01:00:00+01:00 Modified: 2022-12-21T01:00:00+01:00 --- ![Customer First Order Segmentation using Looker and Google BigQuery — Rittman Analytics](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2022%2F12%2F1752872199752-0zj81f.png%3Ffit%3D2500%252C3395%26ssl%3D1) # Customer First Order Segmentation using Looker and Google BigQuery [Looker](https://rittmananalytics.com/blog/category/Looker)[Data Analytics](https://rittmananalytics.com/blog/category/Data+Analytics) 21 Dec Written By [Mark Rittman](https://rittmananalytics.com/blog?author=5846da08f5e231d83261e221) Along with [Recency, Frequency and Monetary Value (RFM) segmentation](https://markr320.sg-host.com/blog/2021/06/20/rfm-analysis-and-customer-segmentation-using-looker-dbt-and-google-bigquery/), one of the most valuable and actionable ways that you can segment your customers is by the first, last and most frequent products and services they order. The product a customer first orders from you is usually a good predictor of customer persona and, for example, their likelihood to go on and make more purchases. The amount a customer spends on their first order is often a good predictor of the amount they’ll spend with you over their lifetime. Channels that bring in customers who over time become your most valuable customers are the channels you should focus your future marketing budget on. We use first and last order customer segmentation to help us make decisions around which sales channels to invest marketing budget on, which products to base our technology recommendations on and in predicting the future lifetime value of new clients, as shown in the example Looker dashboard below. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872199752-0zj81f.png%3Fresize%3D3021%252C4102%26ssl%3D1) Segmenting your customers by first order in a tool such as Looker is a relatively simple process if you’re comfortable with SQL and Looker development, as it will typically involve adding two additional LookML views into your Looker project to calculate: 1. For each customer, the first and last order values for order elements that you’re interested in – product category ordered, sales channel for that order, etc 2. For each transaction, the months, quarters and weeks that the transaction date was from that customer’s first order My usual method for creating these two LookML views is to first create a Looker query that returns all customer orders, one row per order, with all of the fields you want to segment by and also the customer and transaction keys, so that we can join the final LookML views back into our LookML model at the end. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872199447-rureyj.png%3Fresize%3D3470%252C1152%26ssl%3D1) Then I’d click on the SQL button in the explore Data panel to show the SQL that Looker generated, which would typically look like the example below for a Google BigQuery data warehouse data source: `` SELECT companies_dim.company_pk AS companies_dim_company_pk, (DATE(deals_fact.deal_closed_ts)) AS deals_fact_deal_closed_date, deals_fact.deal_source AS deals_fact_deal_source, deals_fact.deal_partner_referral AS deals_fact_partner_referral, deals_fact.deal_pricing_model AS deals_fact_pricing_model, ... FROM `analytics.companies_dim` AS companies_dim LEFT JOIN `analytics.deals_fact` AS deals_fact ON companies_dim.company_pk = deals_fact.company_pk WHERE (deals_fact.pipeline_stage_closed_won ) GROUP BY 1,2,3,4,5... `` To turn this into a query that calculates the first order value for those columns for each customer, I’d use the FIRST\_VALUE () OVER () analytic function to return those first order values and then wrap the query results in a GROUP BY aggregation to return just one row per customer, after first removing transaction date from the results, like this: `` WITH deals AS (SELECT companies_dim.company_pk AS companies_dim_company_pk, first_value(deals_fact.deal_source) over (partition by companies_dim.company_pk order by deals_fact.deal_closed_ts) AS is_first_deal_license_referral, first_value(deal_partner_referral) over (partition by companies_dim.company_pk order by deals_fact.deal_closed_ts) AS is_first_deal_services, first_value(deal_pricing_model) over (partition by companies_dim.company_pk order by deals_fact.deal_closed_ts) AS is_first_deal_dbt, ... FROM `analytics.deals_fact` AS deals_fact JOIN `analytics.companies_dim` AS companies_dim ON deals_fact.company_pk = companies_dim.company_pk WHERE deals_fact.pipeline_stage_closed_won ) SELECT * FROM deals GROUP BY 1,2,3,4... `` Taking the same initial order-level Looker query and then reducing it to just the order key and transaction date, I can then use the FIRST\_VALUE analytic function along with the BigQuery DATE\_DIFF function to calculate months, quarters and years since each customers’ first order date: `` with transactions as (SELECT deals_fact.deal_pk, first_value(deal_closed_ts) over (partition by company_pk order by deal_closed_ts) as first_deal_closed_ts, date_diff(date(deal_closed_ts),first_value(date(deal_closed_ts)) over (partition by company_pk order by deal_closed_ts),MONTH) as months_since_first_deal_closed, date_diff(date(deal_closed_ts),first_value(date(deal_closed_ts)) over (partition by company_pk order by deal_closed_ts),QUARTER) as quarters_since_first_deal_closed, date_diff(date(deal_closed_ts),first_value(date(deal_closed_ts)) over (partition by company_pk order by deal_closed_ts),YEAR) as years_since_first_deal_closed FROM `analytics.deals_fact` AS deals_fact WHERE deals_fact.pipeline_stage_closed_won ) SELECT * FROM transactions ;; `` Note that for this second SQL query, we do not need to aggregate the rows it returns as we want to join it back into our Looker model at the individual transaction level. Finally I then create two new LookML views based on these queries as the source for their derived table SQL, and then join them back to the existing LookML customer and transaction views within in my LookML model at the individual customer and transaction level, like this: ` join: customer_first_deal_cohorts { view_label: " Sales (Hubspot)" sql_on: ${deals_fact.deal_pk} = ${customer_first_deal_cohorts.deal_pk};; type: inner relationship: one_to_one }` join: customer\_first\_order\_segments { view\_label: ” Companies” sql\_on: ${companies\_dim.company\_pk} = ${customer\_first\_order\_segments.companies\_dim\_company\_pk} ;; type: left\_outer relationship: one\_to\_one } .. and now I can start creating my Looker dashboard visualisations. Most of the charts are a variation on the one below which charts the average spend for each segment of customers indexed by quarters since their first order. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872199382-ylxli6.png%3Fresize%3D1324%252C1044%26ssl%3D1) All of these visualizations use a set of common Looker table positional and running total calculations such as the ones in the example screenshot below, with each individual visualisation using particular selections of those calculations. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872199433-3v9xa.png%3Fresize%3D1786%252C1230%26ssl%3D1) For example, another variation on this visualization are the Client decay visualizations in the main dashboard at the start of the article where the initial 100% population of each segment at the time of their first order is then recalculated each quarter after that first order to show the % still active (i.e. have ordered) after that time, using a table calculation such as: `(${companies_dim.count} / index(${companies_dim.count},1))` And finally, if you’re wondering about the final final visualization on the dashboard showing customer spend on projects at the start and end of their engagement with us, it uses Sankey visualization type that you can install from the Looker Marketplace – useful also for customer journey-type data visualizations. ## INTERESTED? FIND OUT MORE Rittman Analytics is a boutique analytics consultancy specializing in the modern data stack who can help you [centralise your data sources](https://markr320.sg-host.com/data-centralization), [optimize your marketing activity](https://markr320.sg-host.com/#:~:text=OUR%20SOLUTIONS-,Marketing%20Attribution,-Modern%20Data%20Stack) and [enable your end-users and data team](https://markr320.sg-host.com/data-team-enablement) with best practices and a [modern analytics workflow](https://markr320.sg-host.com/getting-started-with-dbt). If you’re looking for some help and assistance analyzing and understanding your customer behaviour, or to help build-out your analytics capabilities and data team using a modern, flexible and modular data stack, [contact us now](https://calendly.com/markrittman/initial-discovery-call-with-mark-rittman?utm_source=medium.com&utm_medium=blog&utm_campaign=customer_segmentation) to organize a 100%-free, no-obligation call — we’d love to hear from you! [Looker](https://rittmananalytics.com/blog/tag/Looker)[eCommerce Analytics](https://rittmananalytics.com/blog/tag/eCommerce+Analytics)[Segmentation](https://rittmananalytics.com/blog/tag/Segmentation)[BigQuery](https://rittmananalytics.com/blog/tag/BigQuery)[Data Visualization](https://rittmananalytics.com/blog/tag/Data+Visualization)[Tips & Techniques](https://rittmananalytics.com/blog/tag/Tips+%26+Techniques) [Mark Rittman](https://rittmananalytics.com/blog?author=5846da08f5e231d83261e221) CEO of Rittman Analytics, host of the Drill to Detail Podcast, ex-product manager and twice company founder. [https://rittmananalytics.com](https://rittmananalytics.com/) --- # Modern Data Stack Healthcheck Service from Rittman Analytics Canonical: https://rittmananalytics.com/blog/2022/12/15/modern-data-stack-healthcheck-service-from-rittman-analytics URL: https://rittmananalytics.com/blog/2022/12/15/modern-data-stack-healthcheck-service-from-rittman-analytics Description: Looking back at the past year, a common request we had from new clients was to perform a healthcheck of their modern data stack and to provide enablement for their… Author: Mark Rittman Published: 2022-12-15T22:49:00+01:00 Modified: 2022-12-15T22:49:00+01:00 --- Looking back at the past year, a common request we had from new clients was to perform a [healthcheck of their modern data stack](https://markr320.sg-host.com/services/modern-data-stack-healthcheck/) and to provide enablement for their data team. The healthchecks have been for a number of different organisations and had a different choice of data stack components, for example Fivetran or Airbyte for data ingestion, Looker or Metabase for the BI tool or any number of different cloud data warehouse platforms. One healthcheck was for a data stack where users just didn’t trust the numbers they were seeing in their dashboards, another was for an implementation project where the team didn’t have anyone to lead it with experience in building and leading a data function. Typically the [top ten reasons we see data stack projects going wrong](https://markr320.sg-host.com/blog/2022/12/06/10-ways-your-modern-data-stack-project-can-fail/) are generally one of: - Misalignment with business goals, or not actually solving a business problem - Lack of experience in the tools and techniques required to build a modern data stack - Lack of trust in the numbers being presented to users Our [Modern Data Stack Healthcheck service](https://markr320.sg-host.com/services/modern-data-stack-healthcheck/) gets to the detail of how you’re currently using and building your data stack, and compares it to industry best-practices. Together we’ll evaluate whether or not your investment has delivered the value it should have, create an action plan and work with your team to make your data stack world-class. Over the course of two weeks, our team will conduct a thorough review of your modern data stack, understanding and reviewing it against the original and current goals for your business and auditing the technology, data, processes, and user experience. During those two weeks we cover: 1. Business Goals and Objectives: We undertake an end-to-end review of the design, implementation and use of your data stack, and compare the direction of your data team with your organisational goals. 2. Data Audit and Technical Review: We carry out a thorough review of people, processes and technologies (such as Looker, Snowflake, dbt and other products in the modern data stack ecosystem) in your organization, and identify areas of excellence and areas of weakness. Our team are certified dbt, Looker and vendor tool developers, and have worked on significant numbers of successful data stack implementations – and know also where they can go wrong. 3. Recommendations and Delivery Plan: You’ll receive a written report with clearly assessed scoring covering systematic processes, appropriateness of technologies and their alternatives, and independent review of people covering technical, business, people and leadership skills. We support the review with clear actionable steps to help ensure that weaknesses are addressed through training and prioritisation. … and if you’re coming to the end of your financial year with budget remaining but aren’t yet ready for us to come in? No problem – we can invoice now and schedule for later next year, just let us know when you speak to us. ### INTERESTED? FIND OUT MORE Rittman Analytics is a boutique analytics consultancy that specializes in analytics engineering and the modern data stack. We’ve successfully implemented more than 50 modern data stacks for clients in the UK, USA and Europe, are a dbt Preferred Consulting Partner and blog regularly on modern data stack topics at [https://rittmananalytics.com](https://rittmananalytics.com/blog/). You can [find-out more about our Modern Data Stack Healthcheck service](https://markr320.sg-host.com/services/modern-data-stack-healthcheck/) on our website, download our ebook on [10 Ways your Modern Data Stack Project Can Fail](https://info.markr320.sg-host.com/download-10-ways-your-modern-data-stack-can-fail?utm_campaign=mds_healthcheck&utm_medium=email&utm_source=blog&swcfpc=1&_ga=2.6832114.1952358646.1671114670-1892930500.1664924043) and [contact us now](https://calendly.com/markrittman/initial-discovery-call-with-mark-rittman) to discuss your particular needs and data stack concerns. We’d love to hear from you! --- # 10 Ways Your Modern Data Stack Project Can Fail Canonical: https://rittmananalytics.com/blog/2022/12/06/10-ways-your-modern-data-stack-project-can-fail URL: https://rittmananalytics.com/blog/2022/12/06/10-ways-your-modern-data-stack-project-can-fail Description: The rise of the modern data stack has made it easier, cheaper and faster-than-ever for businesses to centralize their data. Built on cloud-based services, the modern data stack enables businesses… Author: Mark Rittman Published: 2022-12-06T19:41:00+00:00 Modified: 2026-07-02T00:30:00+01:00 --- ![10 Ways Your Modern Data Stack Project Can Fail](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F5b929-1nuyiqegewmps-pizpl7qka.png) The rise of the modern data stack has made it easier, cheaper and faster-than-ever for businesses to centralize their data. Built on cloud-based services, the modern data stack enables businesses to provide reports and KPI dashboards for business end-users, activating that data to improve the efficiency and effectiveness of their customer interactions. Over the past few years, we’ve engaged with many organizations implementing modern data stack projects who were in the “discovery” or “adoption” phase of their journey. Typically they were using services such as Fivetran, Snowflake and Looker to centralize one or two data sources in a cloud data warehouse and to build dashboards for use by “early-adopter” departments and end-users. Those businesses are now moving through the adoption phase and are experiencing common concerns. These include: - Data quality - Data velocity - Data architecture We have also noticed that businesses are struggling to use their data to hit business goals and objectives, due to a lack of clarity and direction of their data stack projects. With ever-increasing cloud infrastructure costs, the modern data stack could well be entering the “trough of disillusionment” stage in the classic “hype cycle” diagram of technology adoption. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F5b929-1nuyiqegewmps-pizpl7qka.png) So what are the main ways that your data projects can go wrong and turn your modern data stack into an expensive failure? I asked our delivery team for the top ten reasons for project failure that they’d managed to help organizations avoid: ### 1\. Unreliable Data that Users Don’t Trust Everyone’s got a new dashboard but they still don’t trust the numbers. The modern data stack makes it easy to replicate your SaaS data into a warehouse and build dashboards for users, but how do you guarantee that all of the data is correct and keeps your users trust as they use it for decision-making? The answer is to make testing central to your analytics development workflow, and automate that testing so that it becomes embedded in how your data team works. Testing of assumptions about new data sources, testing of any new features and functionality provided to end-users and testing of data once it’s in production, observing the shape and skew of data that passes all of your validity checks but is clearly implausible in real-life. ### 2\. Data Centralization too Complex to Scale As new data sources are added to your warehouse, centralising your data becomes more complex. When none of your systems are the definitive source of customer, product, sales or other key business entities, your design approach of working it out as you go along can quickly leave you with a data model that’s complex, confusing and grinding your project to a halt. Scaling your data centralization project beyond a trivial amount of sources needs your team to have a set of design patterns and a layered warehouse design such as the [one we use on our client projects](https://rittmananalytics.com/blog/2020/05/27/2020-5-28-introducing-the-ra-warehouse-dbt-framework-how-rittman-analytics-does-data-centralization/) that: - combines identity across multiple systems when none of those systems are the definitive source of that data - deduplicates multiple sources of customer data even from within the same source system - keeps the velocity and agility of project delivery consistent even as complexity increases - and most importantly. makes sure that numbers are trusted, tested and actually add up properly3. Pace of Delivery — keeping up with the speed of data ### 3\. Pace of Delivery that’s too Slow for the Business A new data stack is of little value when it is unable to provide the data you need — when you need it. And, as your business scales, that demand for more data will only grow stronger. The pace of delivery for your data team needs to match the velocity of your business otherwise it’ll quickly become irrelevant, and you’re back to the bad old days of decisions being taken by the “loudest person in the room”. Addressing this issue of project velocity requires three things to be in place: 1. Data team leadership with experience and an ability to communicate clear and achievable goals 2. An analytics delivery approach that delivers “right first time” at a predictable and rapid pace 3. Access to design patterns, toolkits and accelerators that help your data team complete the simple tasks quickly, freeing them up to focus on the more complex problem solving tasks ### 4\. Misalignment with Business Goals Just as fatal to the success of your data stack are projects that deliver but which are misaligned with your business goals. However hard the data team works and regardless of how innovative their approach, a data team and data stack that are falling short of solving real business problems will struggle to drive their business forward. To avoid this issue depends upon two things; making sure that your analytics team and data stack are consistently being used to solve complex business problems that deliver near-term, quantifiable and tangible benefits for a key, influential stakeholders; and by ensuring that your pace of delivery is sufficient with a focus on solving enough of a problem to get something of value delivered, rather than satisfying technical curiosity. ### 5\. Lack of Experience with new Tools, Technologies and Emerging Best Practices Building a modern data stack requires new tools, technologies, and processes but a central challenge to you is that your new data team lacks experience in this environment. There just aren’t many Heads of Data available for hire who have built a modern data stack, they’re expensive and right now, it’s unlikely you’d get sign-off for a new full-time hire at this level at all. A better approach is to bring in a consulting partner such as Rittman Analytics who have worked with and built data stacks for hundreds of venture-funded, fast-growing businesses. Your consulting partner should also be deeply plugged into the networks of the innovators in the space in order to provide your data team with the context and understanding they need of exactly how these tools work. At Rittman Analytics, we’re typically approached after A and B-series funding rounds, or when a modern data stack project gets the go-ahead in a more established business, and we usually work directly with the CTO and executive team. We set up analytics infrastructure and processes and build the core warehouse, metrics and reports the business needs. As our clients’ businesses grow, we typically transition to more of a supporting role, helping clients grow their data teams by recruiting, interviewing, and training new team members on best practices and our delivery approach. ### 6\. Lack of Planning for Support and User Enablement Smart data teams enjoy learning and solving new problems; building your new data stack is one of the most interesting and rewarding projects your team can work on right now. But what do you do when they move on, taking their new skills away with them to another challenge or opportunity? Who do you have left to maintain and extend the data stack that is so critical to the growth of your business? Consulting partners such as Rittman Analytics address this need by providing ongoing support for data stacks that they or your team built. This is typically structured through a monthly package of hours and services that maintain and monitor your analytics infrastructure. Your consulting partners should also be able to help you identify and resolve any issues caused by source data changes, giving you the reassurance that everything is in safe hands. ### 7\. Cloud Infrastructure Costs getting Out-of-Control Back in the old days of on-premises data warehouses, you knew when it was time to start looking at your query costs when you couldn’t fit any more servers in the server room. Today, the only limit to how many Snowflake credits or BigQuery slots your data stack’s queries can consume is the limit on your company credit card, and tuning and optimizing the SQL generated by your BI tools and transformation processes requires specialist skills. Keeping your cloud infrastructure costs under-control comes down to three things: 1. A development process that considers the cost of ownership of your data pipeline and analytics platform and tests that it runs within acceptable cost parameters 2. A monitoring and altering process that identifies queries and pipeline processes that exceed expected cost limits 3. Experience within the delivery team to tune and optimize the SQL running in your data warehouse ### 8\. Data warehouse platforms that don’t scale Of course, the reason nobody wants to be a DBA today is that with Snowflake, BigQuery and managed service Postgres databases, you don’t need one as they’re largely self-managing. No partitioning needed for Snowflake, no sizing needed for BigQuery, and you can even run your transactional and ad-hoc query workloads on the same Postgres database. The challenge comes, however, when your data stack scales to the point where this hands-off, “it-just-works” approach reaches its limit and the only choice is to throw more and more money at the problem … or accept that this is the limit of how far it will scale. Or you could work with a partner such as Rittman Analytics who have experience working with database technologies such as Firebolt, designed to work economically and efficiently at-scale. All database technologies have “sweet-spots” and scenarios that they optimize for, and what you need is a partner on your team who knows when your current approach has reached its limits, what alternatives would provide a better solution and what the new trade-offs of that solution would be. ### 9\. Data Pipelines that are always breaking Your data stack is a success until Engineering releases a new version of your back-end application and your data pipeline breaks. Something that happens seemingly every few weeks as your engineering and data teams aren’t aligned and the resultant outages mean that vital dashboard KPIs aren’t updated for hours, even days. Solving this problem is part process, and part technology. Regular communications and a productive relationship between your data and engineering teams can help identify and resolve potential issues in the future before they impact on your dashboard availability. Thorough testing and alerting, sandbox and staging environments as well as new concepts such as data contracts can provide smart and thorough solutions as a back-stop. ### 10\. You Built it … but they didn’t come. Perhaps the worst outcome for a modern data stack project is when it’s built as planned, but fails to get the level of user adoption you’d hoped for. A self-service BI project where delivery of new dashboards and reports are still bottlenecked and delivered by a small central team; or the worst fate of all, when your new expensive BI tool is only used to export data to Microsoft Excel! The reality is that building your warehouse and doing the technical work is only the end of the beginning, and not the beginning of the end. Unless you plan now for enabling your end-users to use the platform you’ve built, and activating your data to innovate and transform your business, a strategy of “built it and they will come” will most likely lead to an expensive failure. ### Is Your Data Stack ready for 2023? Now you’ve seen some of the most common pitfalls, why not [get in touch](https://calendly.com/markrittman/initial-discovery-call-with-mark-rittman) with us for a deep dive into how well your Modern Data Stack is performing? Rittman Analytics is a boutique analytics consultancy that specializes in analytics engineering and the modern data stack. We’ve successfully implemented more than 50 modern data stacks for clients in the UK, USA and Europe, are a dbt Preferred Consulting Partner and blog regularly on modern data stack topics at [rittmananalytics.com/blog](https://rittmananalytics.com/blog) . ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F642f8-0ygvkkxse4sfmdwwx.png) *Originally published at* [*https://rittmananalytics.com*](https://rittmananalytics.com/blog/2022/12/06/10-ways-your-modern-data-stack-project-can-fail/) *on December 6, 2022.* --- # Improving Website Search Keyword Performance — Rittman Analytics Canonical: https://rittmananalytics.com/blog/2022/12/02/improving-website-search-keyword-performance-rittman-analytics URL: https://rittmananalytics.com/blog/2022/12/02/improving-website-search-keyword-performance-rittman-analytics Description: Improving Website Search Keyword Performance using Looker, Google Search Console and Fivetran Marketing Analytics 2 Dec Written By Mark Rittman A few weeks ago when checking our company website stats in… Author: Mark Rittman Published: 2022-12-02T01:00:00+01:00 Modified: 2022-12-02T01:00:00+01:00 --- ![Improving Website Search Keyword Performance — Rittman Analytics](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2022%2F12%2F1752872220486-8r000i.png%3Ffit%3D2451%252C1122%26ssl%3D1) # Improving Website Search Keyword Performance using Looker, Google Search Console and Fivetran [Marketing Analytics](https://rittmananalytics.com/blog/category/Marketing+Analytics) 2 Dec Written By [Mark Rittman](https://rittmananalytics.com/blog?author=5846da08f5e231d83261e221) A few weeks ago when checking our company website stats in Looker, I noticed that total sessions had dropped significantly since we [updated it](https://markr320.sg-host.com/blog/2022/08/07/a-new-look-for-rittman-analytics/) back in August. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872220486-8r000i.png%3Fresize%3D2451%252C1122%26ssl%3D1) Visitor numbers were down too and my first thought was that something about the design, usability or changes to content we may have introduced were putting-off visitors. As the vast majority of traffic to our site is anonymous and therefore hard to track across sessions more than a couple of days apart, analyzing visitors by new and returning wouldn’t tell us much about whether visitors were in-fact returning after their first visit. What we could do though is calculate our average page views/session rate across time along with the % of sessions that weren’t classed as “bounce” sessions, both of which indicated that visitors who do make it to the new site are in-fact more engaged, and within those sessions go on to look at more pages on the site than visitors to the previous version. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872220520-8dm3ts.png%3Fresize%3D2451%252C1121%26ssl%3D1) So it’s unlikely to be copy or UX issues driving the drop in traffic, but what about the sources of that traffic? We categorise the various ways by which a visitor arrives at our site into a few standard ways: - **Direct** for when someone types our site URL into their browser address bar, or more likely, when there’s an absence of any other useful information from which we can derive the traffic source - **Social** for when the traffic comes from Twitter, LinkedIn, Medium or another social network - **Organic Search** for when traffic comes from people clicking on our site in search engine results, typically Google Search - **Referral** for when a third-party site, for example segment.com or getdbt.com/partners contains a link to our content - **Email**, for when the visitor clicked-on a link in one of our newsletters Breaking-down those sessions by these traffic acquisition channels showed clearly that, whilst the volume of traffic from referral sources along with email, social and direct had remained at similar levels to before the site update, organic search traffic had cratered to around one-third of its previous level. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872220549-u2hjce.png%3Fresize%3D1600%252C791%26ssl%3D1) This was particularly worrying as just under 40% of the “visitor value” of traffic to our site came search traffic, a [marketing measure we calculate](https://markr320.sg-host.com/blog/2022/08/03/medium-squarespace-or-github-content-marketing-value-analytics-using-looker-dbt-and-segment/) by assigning a score to each category of page on our site that values views of our marketing and services pages higher than ones of our blog and podcast content. A higher visitor value for a session or channel helps us work-out which channels bring us potential new clients vs. ones that are only interested in our technical content right now. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872220715-qy6dz9.png%3Fresize%3D1600%252C671%26ssl%3D1) We’d taken the opportunity to move the site from Squarespace to WordPress when updating its content, and now host it on a GCE VM on our own GCP cloud infrastructure. A big part of the motivation for this move was the opportunity it provided for making use of the CMS data that WordPress stores in an [embedded mySQL database](https://www.wpbeginner.com/glossary/mysql/), which we’d have access to as we were hosting it rather than [running it on wordpress.com](https://wordpress.org/hosting/) or another managed hosting service. The [WordPress database schema](https://codex.wordpress.org/Database_Description) contains all of the page, post and other site content along with the categorisation and taxonomy for that content, details of authors, counts of comments and so on. Adding that data to our [Google BigQuery warehouse](https://markr320.sg-host.com/blog/2022/05/30/how-rittman-analytics-does-analytics-part-2-building-our-modern-data-stack-using-dbt-google-bigquery-looker-segment-and-rudderstack/) and Looker dashboards would give us a canonical reference list of pages on our website that we could join our visitor event data from services such as Rudderstack, Segment, Snowplow and GA4, rather than trying to reconstruct those page details from the context data that those services recorded with page view and other visitor events. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872220781-ne8gp.png%3Fresize%3D1040%252C936%26ssl%3D1) Moreover, if we connected this CMS data from WordPress and the visitor event tracking data we collected via Rudderstack to another data source we’d recently brought into our warehouse, Google Search Console, we could start to try and work out what had changed and what might have gone wrong. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872220715-16kgb5.png%3Fresize%3D980%252C812%26ssl%3D1) Whereas Google Analytics records visits and visitor details from traffic across all sources to your website but can’t tell you why each visit occurred, [Google Search Console](https://search.google.com/search-console/about) allows you to deep-dive into the search activity that comes from Google Searches. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872220714-9xqiwp.png%3Fresize%3D1600%252C1038%26ssl%3D1) The main use marketers and in-particular SEO (search engine optimization) experts have for Search Console data is to understand and then improve the relevance that Google’s algorithm places on your content, firstly by optimizing that content or your overall site to rank higher in “organic” search results. The other approach is where you buy those search impressions and hopefully, click-throughs, by targeting a selection of keywords that historically have brought high-value converting users to your site and bidding for those keywords, a traffic type we categorise as **ppc** (pay-per-click). [The schema provided by Fivetran](https://fivetran.com/docs/applications/google-search-console#schemainformation) for data replicated into your warehouse from the API provided for Google Search Console consists of a number of report-style tables that aggregate your search data by various combinations of site, keyword, page and time dimensions. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872220715-6riijhge.png%3Fresize%3D1600%252C910%26ssl%3D1) In this instance I needed the KEYWORD\_PAGE\_REPORT table that provided the search traffic data of the pages of our site based on the query string user searched for on Google, giving me the ability to see what search keywords were generating traffic for each of our site pages and how our search stats were trending over time. I started by shaping and preparing these datasets initially in Looker, first by creating a LookML view that added page type and category metadata to our WordPress blog, podcast and marketing pages ([code here](https://gist.github.com/markrittman/bd11cff502a92f532c13cacfef6b685a)). Next I took our existing Rudderstack page event data, aggregated it to the week level and transformed the path context variable into a format that could be used to join these page events to the page\_name format that WordPress uses ([code here](https://gist.github.com/markrittman/4687428d095092213b26f06dfaa31b75)), and then aggregated the Keyword Page Report table data to the same week-level ([code here](https://gist.github.com/markrittman/293c80e81cb12d0eecd65bc0b3904fe5)) and then finally, added a new explore to our Looker model to join them all together on the page\_name WordPress page URL slug. I now had a Looker explore that gave me, per week, the volume of traffic we’d received at the site along with the clicks, impressions, average search results position and average click-through-rate for traffic coming to those pages from Google Search users. Over the following days I put together the dashboard below that helped identify the cause of this drop in organic search traffic, fix the issues that caused that drop and now use to make our future marketing activity more relevant and effective. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872221124-fer4ev.png%3Fresize%3D1600%252C1345%26ssl%3D1) The charts and data in the dashboard above show the position we’re in now, but back at the start of November our Looker explore was showing us that both clicks (the number of search users clicking-through to our site from search results) and impressions (the number of times we appeared in search results) were down, suggesting that the drop is likely due to our site content being included in less search results for some reason, which fairly obviously would lead to less clicks through to our site. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872220715-fopolh.png%3Fresize%3D1600%252C732%26ssl%3D1) If we looked at the click-through rate over that period though, it had remained fairly constant. Plotting our average search results position isn’t really a relevant measure at the overall site and keyword level as it includes all the keywords that we don’t optimize our SEO for, where we appear in the search results but as result #50, for example. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872220582-s8rl50w.png%3Fresize%3D1600%252C732%26ssl%3D1) Its when you get down to clicks, impressions and search position measures at the individual search keyword level though, that these Search Console numbers become more relevant. I started by filtering those search stats to just the two weeks that span the old site going offline and the new one coming online, I pivot those two weeks in my explore results and calculate a bunch of variance, % variance and positional measures to create a look that will show me the change in clicks, impressions and average search position as we transitioned to the new site version. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872220588-ugt3qp.png%3Fresize%3D1600%252C942%26ssl%3D1) When we saw the query results in Looker, across the board we could see pretty significant falls in the number of impressions, that, in-turn led to serious drops in click-throughs as users can only click on us if we’re actually in the results of a search. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872220426-e3pu1a.png%3Fresize%3D1600%252C731%26ssl%3D1) So something about the new site had clearly caused caused our content — blog articles, podcast episodes and our marketing and information pages — to disappear off of Google’s search results. And this is where our data-based investigations stopped and we called-in the SEO expert from our marketing agency who we’d brought in to help with some activity coming in the New Year. The problem was indeed our site disappearing from search results and the cause was a combination of factors, primarily around URL formats not matching the ones used on our previous site and the site not having caching enabled and therefore being seen as too slow to provide a suitable user experience for mobile device users. Now we understood why overall traffic had dropped (we’d lost our traffic coming from Google Search results), confirmed that this was the case (individual keyword impressions, clicks and average position stats) and also checked that users who did make it to our site stayed longer, and made more enquiry bookings than before (page view, session and conversion stats from Rudderstack), we knew the problem was an SEO one caused by our site speed and set-up Cloudflare caching and a bunch of other optimizations. The happy-ending to the story can be seen in another Looker visualization as-of today, where we’ve joined the Google Search Console numbers to the page content metadata from WordPress to show the click traffic coming from Google over the past four weeks. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872220715-y3gge.png%3Fresize%3D1200%252C1262%26ssl%3D1) Looking at traffic volume and the amount coming from Google Search we can see an uptick beginning to start mid-way through November when we put the required changes into place, with both impressions and clicks starting to trend back to where they were before the site update and migration. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872220591-g6beni.png%3Fresize%3D1600%252C755%26ssl%3D1) As we start to move out of “damage-rectification” mode and more into “SEO optimization”-mode, this data from Google Search Console going into our warehouse is going to be invaluable in spotting other issues, and identifying opportunities for more effective and relevant marketing. For example, splitting our search keyword click-through-rate into branded (anything with ‘rittman’ in the query keywords) and non-branded (queries such as ‘dbt metrics layer’) shows that we’ve still got some work to do to increase click-throughs on the latter. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872220714-puo0x.png%3Fresize%3D1212%252C914%26ssl%3D1) And finally, putting the set of query keywords that led to clicks to our site into a Looker word cloud visualization, for example, shows us that last week it was interest around open-source alternatives to Segment that brought visitors to our website, an insight into what’s on potential clients’ minds that we can use to better focus our marketing activity in January. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872220427-9wpgr.png%3Fresize%3D1212%252C914%26ssl%3D1) ## INTERESTED? FIND OUT MORE Rittman Analytics is a boutique analytics consultancy specializing in the modern data stack who can help you [centralise your data sources](https://markr320.sg-host.com/data-centralization), [optimize your marketing activity](https://markr320.sg-host.com/#:~:text=OUR%20SOLUTIONS-,Marketing%20Attribution,-Modern%20Data%20Stack) and [enable your end-users and data team](https://markr320.sg-host.com/data-team-enablement) with best practices and a [modern analytics workflow](https://markr320.sg-host.com/getting-started-with-dbt). If you’re looking for some help and assistance understanding the impact and value delivered by your marketing activity, or to help build-out your analytics capabilities and data team using a modern, flexible and modular data stack, [contact us now](https://calendly.com/markrittman/initial-discovery-call-with-mark-rittman?utm_source=medium.com&utm_medium=blog&utm_campaign=dbtsemanticmodel) to organize a 100%-free, no-obligation call — we’d love to hear from you! [SEO](https://rittmananalytics.com/blog/tag/SEO)[Google Search Console](https://rittmananalytics.com/blog/tag/Google+Search+Console)[Looker](https://rittmananalytics.com/blog/tag/Looker)[BigQuery](https://rittmananalytics.com/blog/tag/BigQuery) [Mark Rittman](https://rittmananalytics.com/blog?author=5846da08f5e231d83261e221) CEO of Rittman Analytics, host of the Drill to Detail Podcast, ex-product manager and twice company founder. [https://rittmananalytics.com](https://rittmananalytics.com/) --- # The dbt Semantic Layer, Data Orchestration and the Modern Enterprise Data Stack Canonical: https://rittmananalytics.com/blog/2022/10/21/the-dbt-semantic-layer-data-orchestration-and-the-modern-enterprise-data-stack URL: https://rittmananalytics.com/blog/2022/10/21/the-dbt-semantic-layer-data-orchestration-and-the-modern-enterprise-data-stack Description: We’re currently over in New Orleans for the dbt Labs-sponsored Coalesce 2022 : The Analytics Engineering Conference and I have to say, it’s great to be back at an in-person event… Author: Mark Rittman Published: 2022-10-21T18:40:00+01:00 Modified: 2026-07-02T00:30:24+01:00 --- ![The dbt Semantic Layer, Data Orchestration and the Modern Enterprise Data Stack](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2Fcc3f9-1wt1ysetiwftr6plpf16fpa.png) We’re currently over in New Orleans for the dbt Labs-sponsored [Coalesce 2022 : The Analytics Engineering Conference](https://coalesce.getdbt.com/agenda?ticket=Coalesce+New+Orleans) and I have to say, it’s great to be back at an in-person event since the pandemic lockdowns over the past couple of years. The conversations in the hallways, opinions being shared and friendships built by everyone being in the same place for a few days makes the time and expense involved incredibly worthwhile. Well done to dbt Labs from us and everyone in the community for putting on such a great event. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2Fcc3f9-1wt1ysetiwftr6plpf16fpa.png) There were two big product announcements from dbt Labs on Monday, the first of which was the addition of [Python language support](https://www.getdbt.com/blog/introducing-support-for-python/) into dbt Core. Intended to extend dbt’s appeal to data scientists who typically use Python as their language of choice for building stats and ML models, adding Python language support to dbt and dbt Cloud makes it easier to create data pipelines that involve SQL as well as Python steps and typically run on data lake-style platforms hosted on [Databricks](https://docs.databricks.com/delta/index.html), [Snowflake](https://www.snowflake.com/snowpark/) and [Google Cloud Platform](https://cloud.google.com/learn/what-is-a-data-lake#section-1). The second announcement was the Public Preview release of the [dbt Semantic Layer](https://www.getdbt.com/blog/frontiers-of-the-dbt-semantic-layer/), a layer of business metadata and metrics definitions that aims to place dbt Labs squarely at the centre of the emerging modern (enterprise) data stack, [sherlocks](https://en.wikipedia.org/wiki/Sherlock_%28software%29#:~:text=Sherlocked%20as%20a%20term,-Advocates%20of%20Watson&text=The%20phenomenon%20of%20Apple%20releasing,Mac%20and%20iOS%20developer%20community.) headless-BI startups [cube.dev](https://cube.dev/) and [metricql](https://metriql.com/) and makes it the direct competitor and alternative to today’s standard in that market, Looker’s LookML-based [Universal Semantic Model](https://cloud.google.com/blog/products/data-analytics/lookers-universal-semantic-model). * * * If you’re familiar with the basic concept of a [BI tool semantic layer](https://en.wikipedia.org/wiki/Semantic_layer) the approach taken by dbt Labs will look familiar; a layer of business logic and terminology mappings that sits on-top of your data warehouse and turns the IT-centric world of tables, joins, columns and SQL functions into a trusted and accessible catalog of business objects that end-users can use to create reports and dashboards. Not everyone is an analytics engineer and a semantic layer is essential in delivering the “data democratization” that BI tool vendors from [Business Objects](https://patents.google.com/patent/US5555403) through to [Looker today](https://cloud.google.com/blog/products/data-analytics/democratizing-data-and-insights-with-google-cloud) promise end-users. ![](https://cdn-images-1.medium.com/max/800/0*_Waci2dLP-ED8ZCA) dbt Semantic Layer conceptual diagram, courtesy of dbt Labs The idea behind the dbt Semantic Layer is to create a layer of infrastructure that contains a versioned store of business metrics, along with their dimensionality and rules for time-series and other calculations expressed using a query such as the one below: ``` select *from {{ metrics.calculate( metric('dbt_cloud_weekly_active_users'), dimensions=['country'], grain='day') }} ``` This catalog of metrics and calculation rules is accompanied by infrastructure that crucially, rather than being closed and tied to one particular BI tool is instead made available to the ecosystem of BI tools and other data consumers within the modern data stack. What this in-effect does is unify the data model and understanding of the until-now interoperable, but otherwise silo’d, components of the modern data stack and evolving it into the beginnings of a modern *enterprise* data stack. As such it places dbt Labs in direct competition with Google Cloud Platform and their Looker Universal Semantic Model, an opening-up of the LookML modeling layer to now make it consumable by a number of top-tier enterprise-class data visualization tools including Tableau, Microsoft PowerBI and Google’s own Google Data Studio, now renamed as Looker Data Studio. * * * So how does dbt’s Semantic Layer work, both from a technical level and also commercially? How does dbt, a traditionally developer-centric toolkit designed for batch-transforming data in the warehouse before users get to see it, turn itself into a language capable of expressing complex analytical queries and a query engine capable of supporting fast, ad-hoc analysis? The dbt Semantic Layer consists of a number of components, as shown in the diagram below from the product keynote. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F54573-1kzlorxdvgilicqs0qxfzqg.png) dbt Semantic Layer architecture diagram, courtesy of dbt Labs 1. Metric definitions, created in your dbt project using the new [metrics:](https://docs.getdbt.com/docs/build/metrics) extension to dbt Core and alongside all of your existing data model definitions and transformation logic, that specifies the metric name, type, dimensionality, time grain and filters that can be applied to that metric \# models/marts/product/schema.yml version: 2 models: - name: dim\_customers ... metrics: - name: rolling\_new\_customers label: New Customers model: ref('dim\_customers') description: "The 14 day rolling count of paying customers using the product" type: count\_distinct sql: user\_id timestamp: signup\_date time\_grains: \[day, week, month, quarter, year\] dimensions: - plan - country filters: - field: is\_paying operator: 'is' value: 'true' - field: lifetime\_value operator: '\>\=' value: '100' - field: company\_name operator: '!=' value: "'Acme, Inc'" - field: signup\_date operator: '\>\=' value: "'2020-01-01'" meta: {team: Finance} 2\. The [dbt-metrics](https://github.com/dbt-labs/dbt_metrics#tenets) package extension to dbt Core used to defines rules for calculation of the metric including time-series, period-over-period analysis and rolling aggregates and that can, if used in-isolation, be used to [materialize those calculations](https://github.com/dbt-labs/dbt_metrics#inside-of-dbt-models) for that metric as a static warehouse “metrics layer” table. select \* from {{ metrics.calculate( metric('new\_customers'), grain='week', dimensions=\['plan', 'country'\], secondary\_calculations=\[ metrics.period\_over\_period(comparison\_strategy="ratio", interval=1, alias="pop\_1wk"), metrics.period\_over\_period(comparison\_strategy="difference", interval=1), metrics.period\_to\_date(aggregate="average", period="month", alias="this\_month\_average"), metrics.period\_to\_date(aggregate="sum", period="year"), metrics.rolling(aggregate="average", interval=4, alias="avg\_past\_4wks"), metrics.rolling(aggregate="min", interval=4) \], start\_date='2022-01-01', end\_date='2022-12-31', where="some\_column='filter\_value'" ) }} 3\. Where the magic happens though is in the [new and foundational data infrastructure](https://docs.getdbt.com/blog/understanding-the-components-of-the-dbt-semantic-layer) introduced with the dbt Semantic Layer takes what what is essentially a metrics table generator for developers and turns it into a [ROLAP query engine](https://www.geeksforgeeks.org/difference-between-rolap-and-molap/) designed to support ad-hoc query workloads. A new hosted and always-available dbt Server component (source-available and free for non-commercial use) along with a rewriting SQL proxy server does this by accepting query requests for dbt metrics and live-compiles them into native SQL en-route to your data warehouse, limited initially to Snowflake but with Google BigQuery and Redshift support coming soon. 4\. Finally, the Metadata API component provides access to the Semantic Layer to those BI tools along with ones for data integration, data observability and data science that [announced integration](https://www.getdbt.com/product/semantic-layer-integrations) with the Semantic Layer, either by importing metrics it contains into their own data catalogs or by sending queries using those metrics to dbt’s Proxy Server that then translates, on-the-fly, those queries into native SQL queries against the data warehouse. * * * dbt Labs won’t necessarily be able to translate their success in owning the modern data stack transformation layer into doing so for the semantic layer; there are already established players in that space such as Cube.dev, who already have a working implementation of their product compared to a Public Preview and have extended their product offering to include caching and features aimed at the embedded analytics market. And a semantic model of this type has two fundamental assumptions; that the semantic model is centralized and is owned by your development and engineering team, not by the business users who use it and have the domain knowledge to make it relevant and useful. But fundamentally, what this move by dbt Labs does is divide the modern data stack world into two rival “world views”, not unlike the competition for ideas and influence today between the US and China; one is for a well-planned, efficient but tightly-controlled semantic model that works-well today but is ultimately there to drive more sales of Looker, and depends on Google continuing to invest in it (and not kill it off). The other worldview, the US world view if you follow my tortured analogy, is a coalition of the willing that’s centred around dbt Labs, is an open alliance of BI tool and other vendors that choose to invest in an open standard that, should it succeed, raises the collective value of the modern data stack ecosystem and enables its transition from a disruptive technology only appealing to early adopters into a serious competitor to the monolithic BI Suites that I used to write [1000–page books about](https://www.amazon.co.uk/Oracle-Business-Intelligence-Developers-Guide/dp/0071798749). The significance of dbt Labs adding support for Python workloads into dbt Core is the move it represents into the data orchestration market, and the transformation it represents in the value and fungibility in the product they sell now, dbt Cloud. Right now, dbt Labs’ [valuation at $4.2bn](https://www.prnewswire.com/news-releases/dbt-labs-raises-222m-in-series-d-funding-at-4-2b-valuation-led-by-altimeter-with-participation-from-databricks-and-snowflake-301489733.html) on [revenue of around $60m](https://growjo.com/company/dbt_Labs) comes largely from sales of their dbt Cloud service to large enterprise customers whose list of regulatory needs they need to comply with is matched only by the deepness of their pockets and willingness to pay for it, but also have the IT and engineering support that can build their own schedulers and release pipelines out of stock AWS or GCP cloud services. Adding “good-enough” data orchestration capabilities into dbtCloud will, over time, make it a compelling alternative to paying for products such as Dagster ([Elementl](https://www.elementl.com/)), Airflow ([Astronomer](https://www.astronomer.io/)) and [Prefect](https://www.prefect.io/) when budgets are under pressure, and the extension in user personas served that it represents increases the potential audience for users of the service. More significantly though, the dbt Semantic Model is well-placed to become the universal semantic model and unifying infrastructure for the modern data stack as that ecosystem of products, like most disruptive technologies, extends its appeal from early adopters up into the enterprise space. And being the software company that realised that vision and building the software that enables and is essential in delivering that vision and making it relevant to the enterprise market is what’s presumably the driver for the 70x multiple that dbt Labs current revenue represents on that $4.2bn valuation. ### Interested? Find out More Rittman Analytics is a boutique analytics consultancy specializing in the modern data stack who can help you [centralise your data sources](https://rittmananalytics.com/data-centralization) and [enable your end-users and data team](https://rittmananalytics.com/data-team-enablement) with best practices and a [modern analytics workflow](https://rittmananalytics.com/getting-started-with-dbt). If you’re looking for some help and assistance understanding the impact and value delivered by your content or other marketing activity, or to help build-out your analytics capabilities and data team using a modern, flexible and modular data stack, [contact us now](https://calendly.com/markrittman/initial-discovery-call-with-mark-rittman?utm_source=medium.com&utm_medium=blog&utm_campaign=dbtsemanticmodel) to organize a 100%-free, no-obligation call — we’d love to hear from you! --- # Join us at Coalesce2022, New Orleans and London this comingweek! Canonical: https://rittmananalytics.com/blog/2022/10/16/join-us-at-coalesce2022-new-orleans-and-london-this-comingweek URL: https://rittmananalytics.com/blog/2022/10/16/join-us-at-coalesce2022-new-orleans-and-london-this-comingweek Description: The Rittman Analytics team will be at Coalesce 2022, the analytics engineering conference sponsored by our partner dbt Labs, at the New Orleans in-person event running all this week in… Author: Mark Rittman Published: 2022-10-16T09:00:00+01:00 Modified: 2026-07-02T00:35:54+01:00 --- ![Join us at Coalesce2022, New Orleans and London this comingweek!](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2Fcbb25-0eq0bvpqk8cs2deu7.png) The Rittman Analytics team will be at [Coalesce 2022](https://coalesce.getdbt.com/?utm_term=search-keywords&utm_campaign=fy-2023_coalesce-2022_awareness_search&utm_source=google&utm_medium=paid-search&utm_content=605596583219&hsa_acc=8253637521&hsa_cam=17498466860&hsa_grp=137663559283&hsa_ad=605596583219&hsa_src=g&hsa_tgt=kwd-1504390971839&hsa_kw=coalesce%202022&hsa_mt=e&hsa_net=adwords&hsa_ver=3&gclid=Cj0KCQjw166aBhDEARIsAMEyZh4QWwn6BgjbQ0Xep8Ps157-OFdGuJ2n3RE6mngboLzqh0zfaMvD0gMaAqEWEALw_wcB), the analytics engineering conference sponsored by our partner dbt Labs, at the New Orleans in-person event running all this week in [New Orleans, USA](https://coalesce.getdbt.com/new-orleans) and at the [London, UK Finale](https://coalesce.getdbt.com/online-london) on Thursday October 20th. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2Fcbb25-0eq0bvpqk8cs2deu7.png) I’m joined in New Orleans by [Lewis Baker](https://medium.com/@lewischarlesbaker), our Head of Delivery along with [Olivier Dupuis](https://www.linkedin.com/in/olivierdupuis/?originalSubdomain=ca) and we’ll be at the Partner Day on Monday afternoon for an update on what’s next for dbt Labs and the Partnership Program. We’ll be around for the rest of the week so feel free to stop us and talk analytics, it’ll be great to be at our first in-person conference since the pandemic. Jordan Ilyat and our first sales hire, [Ben Zarif](https://www.linkedin.com/in/ben-zarif-b9b45a110/), will be at the London event this Thursday so again, if you’ve got an analytics story to tell or if we can help in any way, stop us and say hello or drop me an email at [mark.rittman@rittmanalytics.com](mailto:mark.rittman@rittmananalytics.com) — and hopefully see some of you in New Orleans or London! --- # Behavioural Analytics — Querying fast and slow Canonical: https://rittmananalytics.com/blog/2022/10/14/2023-3-1-behavioural-analytics-querying-fast-and-slow URL: https://rittmananalytics.com/blog/2022/10/14/2023-3-1-behavioural-analytics-querying-fast-and-slow Description: This article aims to explore a layman’s approach to behavioural analytics, using concepts within behaviouralism and behavioural science to unlock new ways of using analytics to help us achieve organisational… Author: Mark Rittman Published: 2022-10-14T16:43:00+00:00 Modified: 2022-10-14T16:43:00+00:00 --- This article aims to explore a layman’s approach to behavioural analytics, using concepts within behaviouralism and behavioural science to unlock new ways of using analytics to help us achieve organisational goals. These are complex disciplines. Behaviouralism is philosophic and behavioural science is the scientific¹. Like most philosophical and scientific disciplines it requires a lifetime of study to become truly informed. However, gaining an understanding of the foundations of these two subjects has become increasingly accessible and I believe that with this accessibility, it’s possible to gain a basic level of understanding that can open up **a way of thinking** about analytics that can dramatically improve analytics functions within many organisations. This way of thinking puts understanding human behaviour at the heart of our decision-making, leading us to make decisions and create interventions that nudge end users to make more desirable choices. It’s this way of thinking I want to explore. ## So What Is Behaviourism? The first explicit behaviouralist was John B. Watson “who in 1913 issued a kind of manifesto called Psychology As the Behaviouralist Views It”.¹ What Watson was arguing is that psychology should be redefined as the study of behaviours. This was radically different to the thinking at the time, where most psychologists were predominantly concerned with consciousness and mental processes. This disruption was compounded when in 1971 B.F Skinner published Beyond Freedom and Dignity, which marked a landmark within the paradigm of **radical behaviouralism,** something much more exciting than plain old behaviouralism… This new radical behaviourism saw a future where the often self-destructive and unpredictable nature of humans could be corrected through behavioural conditioning. Skinner writes; “What we need is a technology of behaviour, We could solve our problems quickly enough if we could adjust the growth of the world’s population as precisely as we adjust the course of a spaceship…”² Skinner saw behaviouralism as a way of saving ourselves from ourselves and perhaps the only way to save ourselves. This radical view speaks to his belief in the power of human behaviour not only in our lives as individuals but as a species. Paired with this radical new philosophy, in the coming years another powerful force emerged; the collection and processing of data on a mass scale. Behavioural scientists and commercial interest over this period became increasingly aware of how these two powerful disciplines can be used together. What does this mean to the modern data scientist or analytics engineer? Well, the behaviouralists would probably argue, we are in a position of great power. ## The basics of behavioural science in application to data analysis So far, we have covered a very brief overview of the emergence of behaviouralism, the concepts of which we will apply to our analytics using methodologies that sit within behavioural science. Let’s look into how analytics fits into all of this. ### Beyond freedom and dignity AND descriptive analytics… Broadly speaking, there are three types of analysis: 1. Descriptive 2. Predictive 3. Causal³ Developing a BI platform that provides reliable and fast **descriptive** analytics is no mean feat and can be very powerful. It allows you to understand things that have happened and perhaps why they happened. It is however limited in its ability as it only looks to describe past events. **Predictive** analytics helps you look forward, forecasting future trends to better help decision-making in the present. **Causal** analytics on the other hand offers the opportunity to carve out the path for future activity by modifying the behaviours of end-users. We can use Skinner to provide us with an example of this, he writes: “We tend to say, often rashly, that if one thing follows another, it was probably caused by it — following the ancient principle of **post hoc, ergo propter hoc (**after this, therefore because of this)**.** An example of this is if we speak sharply with a friend and are asked why a reasonable response would be ‘I felt angry’”¹. Anger here is the cognition or emotion and the reaction is the behaviour it invokes. It’s this relationship that casual analytics looks to understand and utilise, which in turn brings us to the question, what can we utilise? Wendel’s summary of research on this topic gives us some useful pointers: - *We’re limited* beings in attention, time, willpower, etc. - *We’re of two minds*: our actions depend on both conscious thought and non-conscious reactions, like habits. - Because of these limitations, our minds use shortcuts to economise and make quick decisions. - Our decisions and our behaviour are *deeply affected by the context* we’re in, worsening or ameliorating our biases and our intention–action gap. - One can *cleverly and thoughtfully design a context* to improve people’s decision-making and lessen the intention–action gap.³ Understanding these behavioural qualities allows us to better approach causal analytics. The acknowledgement that we are limited beings and do not operate on a purely rational basis can change the treatment we give end-users. The fact that we are of two minds, the conscious and the non-conscious can prompt us to think about how our neurology impacts how we structure our business behaviours. Taking into account the mental shortcuts people use to make decisions and how context can also affect these decisions enables us to better align the actions we take to the people we are focused towards. What does this mean to behavioural analysis? If you can develop models and methodologies that take advantage of how individuals are likely to behave, you can nudge them in the direction you want them to go in. If your business has the potential, you can constantly nudge them into the direction you want them to go in and become habit-forming through cues, routines, and rewards.³ With these considerations in mind, you can see how using our knowledge of behavioural science can help us encourage end-users to do certain things for their or our betterment. ## The process for developing behavioural models — A case study To bring theory to life, I have used basic understandings and processes to approach a business question. It will hopefully provide practical context for the concepts and processes we have and will be covering. For our case study, we will be framing it within the bounds of a **behavioural story.** For our behavioural story, we are very lucky as there is a double coincidence of wants. **We want employees to gain certification and employees want to be certified**. From a commercial point of view, it strengthens our ability to deliver and brings us closer to our partners. For the individual, it increases their equity, satisfies their desire to learn and enables them to succeed in their personal development. Despite this fortunate convergence of wants, putting in time for certification in a growing consultancy can be a challenge. The business and individual **want** to do it but billable client work often gets prioritised when utilisation starts to run hot. This leads to the **intention-action gap** where people want to put time aside for certification but it gets pushed aside for seemingly more important activities. This manifests into a behaviour where engineers default to prioritising client work over training. This behavioural story is a sensitive one. Client work pays the bills and strategic messaging and company culture can move in opposite directions, a case of “Culture eats strategy for breakfast”. This being said, there’s nothing wrong with hoping for better and behavioural analytics can be a way of reaching higher levels of self-awareness through empirical evidence. To tackle this behavioural story, we are eventually going to be looking at developing a theory of change (ToC) to intervene with this behaviour but first, we need to work through some procedures to equip ourselves to approach it. The process we will adopt is the one outlined in Behavioural Data Analysis with R and Python by Florent Buisson which is an excellent resource. However, before we can start working through a method for behavioural analysis, we need to figure out what are our variables. ### What data am I using? We collect timesheet data on our staff from Harvest, it’s best used when anonymised and tells us an interesting story about how people spend their time across internal and external activities. We also collect behaviour and sentiment data through a platform called OfficeVibe. These two data sets together allow us to gain insights into how workload, types of work and project concurrency impact people’s emotions and well-being within the workplace. This provides great potential for analysing factors that impact utilisation and workplace sentiment. ### The approach Please note, this section is very much a digest of Behavioral Data Analysis with R & Python by Florent Buisson. I have made some minor modifications to what I interpret their workflow to be but I take no credit for the methods discussed. So far we have explored some of the conceptual foundations and how they can apply to analytics, which seems brilliant! This however leads us to question how we approach it… Luckily Buisson has put forward a great method which looks something like the workflow below: Using this method we can approach analysis in a methodological and process-driven fashion. 1. Our first step is the frame the business goal. We’re going to be looking into the goal to “Increase the level of certification within our business” 2. We then need to identify our key metric to measure this. In our instance, it is “rate of certification against target” 3. We then need to create our causal diagram (CD). There’s a method to doing this. The approach I have taken took three iterations. I first of all mapped out all of the variables I am aware we have data points for. I then looked at correlations between these variables. 4\. Once we have an understanding of the correlations, we can refer back to our CD. By matching our variables with the correlation matrix, we can start to prioritise which variables are likely to allow us the influence our key metric (rate of certification against target) and then rationalise our CD. 5\. Now that we have iterated through our CD and formed some opinions about how to best achieve our goal we can plan our intervention. I found a tight correlation between overall feelings of satisfaction, delivery headcount and the number of hours spent on certification (surprise, surprise). These are levers that I can now pull to try and improve our key metric. To avoid compounding variables, I’m only going to make one change which is to ensure we up headcount. Having a control and intervention group with employees feels a bit mean so we won’t deploy that in this instance… 6\. We now need to monitor the results of our intervention over time. An example of this would be an uptake in certification against a target or a tighter correlation. Steps 1 through 5 have also helped us to establish our theory of change: Using the process and methods above, we can systemically build behavioural models that allow us to better influence the behaviours of end-users to achieve our organisation’s goals ### What did I find out? I found out that there’s a close correlation between feelings of satisfaction and time spent training. It does raise the question, does workplace satisfaction encourage learning or does learning lead to workplace satisfaction or do both lead to each other? Either way, further promoting this emotion is a good lever to pull. It also brought into focus the importance of adequate staffing, yet another lever to focus on. It also got me thinking about other variables I could include in our behavioural logic such as talking about it in our daily stand-ups, something that’s harder to measure. ## HOW THIS APPLIES TO THE MODERN DATA STACK So far, we have established a top-level view of behavioural science and a layman’s approach to it. What is interesting to me is how these concepts and processes could fit within a data warehouse. We are provisioning for the causal layer of the analytics hierarchy by having what we call the methodological layer as part of the warehouse architecture. The architecture above looks to leverage the semantic layer to call out behavioural variables for analysis bringing you higher up the analytic hierarchy. The aim of this would be for causal analytics to fit within a standard warehouse architecture, making use of the ontological layer and semantics to feed behavioural variables into the shape of our causal diagrams. This methodological layer should not be restricted to behavioural data and the aim would be to extend it into multiple disciplines all of which impose methods on business entities, moving us away from over-indexing on descriptive analytics. Future articles may look into using [https://github.com/LewisCharlesBaker/droughty](https://github.com/LewisCharlesBaker/droughty) to provide an interpreter for behavioural grammar. ### The counterargument and the tempering of behaviouralism In recent years, behaviouralism has fallen under scrutiny as behaviouralists commonly used animals in experiments under the assumption that what was learned using animal models could, to some degree, be applied to human behaviour. These assumptions fell when the original research that supposedly validated the arguments of behavioural had consistently different results when repeated. Critics of behaviouralism use this data to validate an important argument against the impact that nudging individuals by methods such as the cue, routine and reward system has when applied. This puts forward the notion that behaviouralism is perhaps scientistic rather than scientific. Furthermore, what tempers the power of behaviouralism is the ability for individuals to make their own choices through a conscious effort which is something the behaviouralists have been seen to overlook. ## In conclusion… The concepts and research within both behaviouralism and behavioural science offer valuable lenses we can use to model, test and act on our data. Even having a rudimentary understanding of these disciplines provides a way of thinking about data that can unlock potential from data that we would otherwise be unable to access. The case study of certification within our company, it’s enabled me to think about how business behaviours impact our employee behaviours and how we can modify these variables in a methodological and controlled way to better achieve our business goals (and most importantly, to help them learn!). ¹ About Behaviorism — Skinner, B.F. ²Beyond Freedom and Dignity — Skinner, B. F. ³ Behavioral Data Analysis with R & Python — Florent Buisson ⁴ How the Mind Works — Steven Pinker ⁵ Designing for Behavior Change: Applying Psychology and Behavioral Economics — Wendel --- # A New Look for Rittman Analytics Canonical: https://rittmananalytics.com/blog/2022/08/07/a-new-look-for-rittman-analytics URL: https://rittmananalytics.com/blog/2022/08/07/a-new-look-for-rittman-analytics Description: If any of you have experience running a business, you’ll know that your own marketing efforts can get put on the back burner while client work takes priority. Well, that… Author: Mark Rittman Published: 2022-08-07T12:00:00+01:00 Modified: 2026-07-02T00:31:25+01:00 --- ![A New Look for Rittman Analytics](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F71776-0e7nvnf9d5xclfzet.png) If any of you have experience running a business, you’ll know that your own marketing efforts can get put on the back burner while client work takes priority. Well, that happened to us too. For the last three years we’ve been working hard to advance the state of analytic engineering while our own website and company branding has received little attention As we reflected on our work with our branding agency [STANCE](https://www.thestancemethod.com/), we didn’t want to simply give ourselves a new lick of paint. We wanted to clarify the circumstances and projects that lead to our best work. (Which, in turn, has led to a new and much needed lick of paint on our website, [https://rittmananalytics.com](https://rittmananalytics.com/)) ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F71776-0e7nvnf9d5xclfzet.png) In this update, I’ll be telling you about the future of the business and how we best plan to serve our clients in the future. 1. Developing a competitive advantage by modernising your approach to data 2. Building data excellence into the heart of businesses 3. Data hungry start-ups and fast-growing businesses 4. Refreshed brand identity 5. The Rittman Analytics blog, and Drill to Detail Podcast **Developing a competitive advantage through data** If you’ve been through a rebrand, you’ll know it involves a bit of navel gazing coupled with insightful chats with your best clients to help you understand what makes you, you! Through anecdotes and evidence, we heard that we help businesses become more competitive through modernising their approach to data analytics. - Monolithic BI suites that couldn’t scale were being replaced by modern, modular cloud-hosted data stacks. - Dashboards and reports that might have taken weeks to create can take minutes after one of our sprints. - Insights that might have been inaccurate or impossible to gather are now trusted and available as self-service to users There’s no question in our mind: We believe businesses who modernise their data stack and the way they deliver insights can develop a true competitive advantage. **Building data excellence into the heart of businesses** Through our branding process, we have crystallised what constitutes data excellence and standardised how to embed it into a project. Our distilled offer helps businesses: - Define their data strategy - Implement a successful modern data stack - Develop a strong data culture ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F840f0-1kbxoi2xsjdstpjaz2quivq.png) Years of experience have taught us the needs of different industries, the data collection nuances of SaaS technologies and how best to structure their data for warehousing and transformation. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2Fac674-1gyo4wujyzgflkrsysxjd7q.png) The upfront definition of a data strategy and wrap-around of a strong data culture are two traits that make a [Rittman Analytics project](https://rittmananalytics.com/about/how-we-work/) even more successful. We’re excited to give them [pride of place on our website](https://rittmananalytics.com/) in service of our mission: to build data excellence, a modern data stack and a mature analytics workflow into the heart of your business. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2Ffbb86-13exdehgmyfm3-udaehk8uq.png) **Data hungry start-ups and fast-growing businesses** After three years in business, [we know a good customer fit](https://rittmananalytics.com/case-studies/) when we see one. They’re typically either high-growth VC-funded startups or data teams within more established businesses looking to adopt the best characteristics of a tech startup, and typically within vertical industries we specialise in, including: What they all have in-common is the huge amounts of data they’re generating full of valuable signals each day, a volume of data that is rapidly outstripping their ability to model, transform and make insights available to the right people, at the right time and in the right format. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F4f419-15ctpjc_niitbuoiu9h5uka.png) It might be the case that their data stack isn’t in the best of shape, is based on legacy technology that can no-longer scale or is missing capabilities that users now demand. It might be that they’re trying to “boil the ocean” by centralising all their data without a clear brief of what “success” looks like. **Refreshed brand identity** As you encounter our new branded assets, we aim to imbue two concepts: - Architectural excellence - Data abstraction What was the world of analytics like before analytics engineers? Messy. As engineers, we are here to build data excellence. You’ll see this in our diagrams and the structure of our site layout. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F7b9d8-1lxgyihovvmz_oszyjxd8lw.png) Yet, no two projects are identical, each with its own data intricacies. You’ll see this brought to life in abstract shapes, showing how we mould to your needs. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F6c0d9-1obw8if9qakdl-9h1iwhjiw.png) Our colour palette adds a little charge and from time to time, you’ll see a grid that might remind you of a circuit board. And we’ve not finished yet … there’s one more thing that’s coming later in the year that we’re even more excited about, and links all of this together. All-in good time though. **The Rittman Analytics blog** One area where we have focused over the years has been the blog. You’ll see many links to it over our site, adding meat to the bones of some of our marketing-friendly content. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F861fc-1tefcxxmbgey5abarfrkp5a.png) Some of our content has become the de-facto resource for all things dbt and we encourage you to check out: - [How Rittman Analytics Does Analytics Part 2 : Building Our Modern Data Stack Using Dbt, Google Bigquery, Looker, Segment And Rudderstack.](https://rittmananalytics.com/2022/05/30/how-rittman-analytics-does-analytics-part-2-building-our-modern-data-stack-using-dbt-google-bigquery-looker-segment-and-rudderstack/) - [Customer Cohorting, Retention Curves And Predictive Lifetime Value Using Looker And Google Bigquery](https://rittmananalytics.com/2021/02/22/customer-cohorting-retention-curves-and-predictive-lifetime-value-using-looker-and-google-bigquery/) - [Ad Spend And Campaign Roi Analytics Using Segment, Looker, Dbt And Google Bigquery](https://rittmananalytics.com/2020/09/19/ad-spend-and-campaign-roi-analytics-using-segment-looker-dbt-and-googlenbspbigquery/) And finally … it’s almost the 100th episode of the [Drill to Detail Podcast](https://rittmananalytics.com/drilltodetail). We’ve got a great episode lined-up with three of our most popular episode guests, and a mystery special guest who I know will really make it an episode to remember. That’s all from me. I hope you like our new website, offer and branding. If you would like to chat about our services, [here is my calendly](https://calendly.com/markrittman/initial-discovery-call-with-mark-rittman?utm_source=medium.com&utm_medium=blog&utm_campaign=new_look_for_rittmananalytics) link . All the best, Mark Rittman CEO Rittman Analytics. * * * *Originally published at* [*https://rittmananalytics.com*](https://rittmananalytics.com/2022/08/07/a-new-look-for-rittman-analytics/) *on August 7, 2022.* --- # Medium, Squarespace (or Github?) : Content Marketing Value Analytics using Looker, dbt and Segment Canonical: https://rittmananalytics.com/blog/2022/08/03/medium-squarespace-or-github-content-marketing-value-analytics-using-looker-dbt-and-segment URL: https://rittmananalytics.com/blog/2022/08/03/medium-squarespace-or-github-content-marketing-value-analytics-using-looker-dbt-and-segment Description: If you’re reading this article on our Medium.com blog blog.rittmananalytics.com, you may not be aware that we also cross-post these blog posts to our own company website, at https://rittmananalytics.com/blog. Our… Author: Mark Rittman Published: 2022-08-03T18:12:00+01:00 Modified: 2026-07-02T00:30:19+01:00 --- ![Medium, Squarespace (or Github?) : Content Marketing Value Analytics using Looker, dbt and Segment](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F6b785-1oggfldngfv6ed5hr2ae2aq.png) If you’re reading this article on our Medium.com blog [blog.rittmananalytics.com](http://blog.rittmananalytics.com/), you may not be aware that we also cross-post these blog posts to our own company website, at [https://rittmananalytics.com/blog](https://rittmananalytics.com/blog). ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F6b785-1oggfldngfv6ed5hr2ae2aq.png) Our two blog sites, on the left hosted by Medium.com and on the right, by Squarespace We cross-post blogs from Medium.com to our own Squarespace-hosted site partly to make our content available across as many channels as possible, but also because we assume that content published on our own site has [more chance of “converting” into a new sales enquiry](https://rittmananalytics.com/blog/2022/5/30/stitching-identity-across-the-customer-journey-using-segment-google-bigquery-and-looker) than content hosted on a third-party site such as Medium.com. But is that assumption correct? Or does an aggregator site such as Medium bring in a different type of audience who have a greater propensity to convert than visitors to our website coming in most cases via organic search? Does Medium.com’s design, readability and ability to recommend our articles to readers with an interest in the topics we write about lead to more of those articles being read through to the final call-to-action that’s at the end of all our blog posts? ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F034ba-1igjxuief3lpbl4nkhgt4pq.png) Call-to-action paragraphs at the end of every blog post As a data-driven analytics consultancy with [our own modern data stack](https://rittmananalytics.com/blog/2022/5/30/how-rittman-analytics-does-analytics-part-2-building-our-modern-data-stack-using-dbt-google-bigquery-looker-segment-and-rudderstack), in this blog I’ll walk through how we took data from the blog on our site collected in this case using Segment along with data from Medium we were able to extract and scrape, how we transformed, standardised and integrated those datasets using dbt and then answered our own question using a series of looks and analyses created using Looker. * * * Medium.com does have a [REST API](https://medium.com/@cndro/how-to-get-data-from-medium-api-ca3b3a660a81) that we could use to extract a list of our articles, publications and counts of article claps (analogous to ‘likes’), but for convenience we use scraping service [PhantomBuster](https://phantombuster.com/) along with [Airbyte](https://airbyte.com/) to first extract the details of our blog posts, post dates and clap counts in CSV form, then load in raw form into our BigQuery data warehouse in a format that reflects the scraping and Medium.com APIs that this tool uses. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F4833c-1dixwyt_enbcsabosdbwhka.png) We actually use two separate tasks, or “phantoms”, to perform this task with the second automatically run on completion of the first; 1. An initial phantom that searches for all posts from Rittman Analytics and returns the post title, URL and aggregated stats for reads, impressions and claps (likes) 2. A second phantom, “chained” to the completion of the first one, that gives us the name and public profile URL of each article’s clappers (likers), but not their email addresses or any personal or contact information. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2Fe46a2-1_prektq9hhomjeinl5b8yw.png) Displaying this data from Medium as a Looker table visualization is a useful start and shows me the overall popularity of the blogs we’ve published on Medium; articles I wrote back in 2018 on [Google BigQuery table partitioning](https://blog.rittmananalytics.com/date-partitioning-and-table-clustering-in-google-bigquery-and-looker-pdts-2bab9ec3be19) and [using Looker to analyze Monzo bank transactions](https://blog.rittmananalytics.com/using-looker-data-actions-to-make-monzo-spend-analysis-more-interactive-and-actionable-43917d34a552) have historically been the most popular … but did this popularity translate into visits to our website and new leads? ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F68ff4-1-94pxd7ggrvxgvumfs3m0w.png) Answering this business question on the outcome and results of our blogs and content requires us to source and derive a further set of data points: 1. Page view and other event data captured on our main website using [Segment](https://rittmananalytics.com/blog/tag/Segment), giving us details of any referral sessions coming over from our Medium site (note that Rittman Analytics is a [Segment Select Channel Partner](https://www.partnerbase.com/rittman-analytics/segment) specializing in [Multi-Touch, Multi-Source Marketing Attribution](https://segment.com/academy/advanced-analytics/an-introduction-to-multi-touch-attribution/)) 2. Some method of scoring that referral traffic and identifying conversions, so that we can work out whether the traffic coming across from Medium is useful from a marketing and lead-generation perspective 3. A way of joining these two datasets together, so that we can link back from that referral traffic back to the data we have on our Medium posts to work out whether claps correlate to project enquiries, and if certain topics work better than others in terms of generating interest in our services. We’re already capturing page view and track event data coming from Segment and storing it in two fact tables in our warehouse, one at event level and another with those events organized into user sessions. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2Fb77b8-1vtwg6mvhg9g7o5el9c0eew.png) We can see what sessions had originated by visitors coming over from our Medium site by filtering on the referrer\_host column in our sessions fact table. SELECT \* FROM \`ra-development.analytics.web\_sessions\_fact\` WHERE referrer\_host IN ('blog.rittmananalytics.com','medium.com/mark-rittman') ORDER BY session\_start\_ts LIMIT 10 Within the LookML view we create over this warehouse table in Looker we add a further virtual column, or “LookML dimension”, that categorises page view events as blog or podcast article views or more interestingly for us, as views of marketing or services pages. dimension: page\_category { group\_label: "Behavior" type: string sql: case when ${event\_type} = 'Page View' then case when ${TABLE}.page\_url\_path like '%blog%' then 'Blog' when ${TABLE}.page\_url\_path like '%drilltodetail%' or ${TABLE}.page\_url\_path like '%podcast%' then 'Podcast' when ${TABLE}.page\_url\_path = '/' or ${TABLE}.page\_url\_path like '%home-index%' then 'Home Page' when ${TABLE}.page\_url\_path is not null then 'Marketing' end end;; } For visitor events coming from our site that aren’t page views we flag those that we’d consider “conversion” events, at the time of writing any events with “Button” in the event name (“Contact Us Button”, “Contact Button”, “Button Pressed” and so on), and create a measure counting those events in order to measure our session and visitor conversion rates. dimension: is\_conversion\_event { type: yesno group\_label: "Behavior" sql: ${event\_type} like '%Button%' ;; } measure: total\_conversions { type: count\_distinct sql: ${web\_event\_pk} ;; filters: \[is\_conversion\_event: "Yes"\] } Finally, all of that information is then combined in another LookML dimension that calculates the **visit value** of each website event, using logic that assigns a value of 1 for other content page views but doubles in-value for other event types that correspond to stages in our lead capture funnel, with a measure that aggregates this visit value in order to then calculate the average visit value for sessions and users. dimension: visit\_value { type: number hidden: yes sql: case when ${page\_category} in ("Blog","Podcast") then 1 when ${page\_category} = "Home Page" then 2 when ${page\_category} = "Marketing" then 4 when (lower(${event\_type}) like '%collateral%' or lower(${event\_type}) like '%pricing%') then 8 when lower(${event\_type}) like '%button%' then 16 end;; } measure: total\_visitor\_value { type: sum sql: ${visit\_value};; } If we then want to plot this value by months since each blog article was posted, we need to find a way to join the article data from Medium with the referrer URL data for each visitor event coming from Segment; the challenge is that that by default those URLs don’t match-up and return rows in a SQL join: 1. Article URLs given to us by Medium for us are in the format [https://blog.rittmananalytics.com/](https://blog.rittmananalytics.com/) and [https://medium.com/mark-rittman/](https://medium.com/mark-rittman/%3Carticle_stub%3E) 2. Within our Segment page event data, Medium uses the URL format [https://medium.com/article/](https://medium.com/article/%3Carticle_stub%3E) regardless of whether the traffic came from our Medium publication or my personal Medium page Working around these mismatching URL formats we extract just the article stub string from the Segment page event referrer column in the relevant LookML view, like this: dimension: referrer\_article\_stub { group\_label: " Acquisition" type: string sql: case when ${referrer\_source} = 'Medium' and ${referrer\_host} = 'blog.rittmananalytics.com' then split(${referrer},'/')\[safe\_offset(3)\] when ${referrer\_source} = 'Medium' and ${referrer\_host} = 'medium.com' then split(${referrer},'/')\[safe\_offset(4)\] end ;; } and then do the same for the LookML dimension that contains the blog article data from Medium, like this: dimension: article\_stub { primary\_key: yes type: string sql: case when ${utm\_source} = 'medium' then split(${post\_url},'/')\[safe\_offset(4)\] when ${utm\_source} = 'squarespace' then ${post\_url} end ;; } We can then create a Looker visualization such as that shown below that displays, over time, the cumulative visit value generated by each Medium blog over the months since each was first published. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F5c9b5-1pybfqqfnhyu4wlmvxfb-rq.png) Calculating growth or “ramps”-style data visualisations such as this requires two more data items: 1. A LookML dimension containing the number of months since a reference date or timestamp that the event or transaction happened on; for our example this is the number of months from the original date of each blog’s original posting that the visitor event happened on (“Weeks Since Blog Post” in the chart’s X axis), something that we can now calculate since we’ve joined our Medium and Segment, like this: dimension: referrer\_days\_since\_post { group\_label: " Acquisition" type: number sql: case when ${referrer\_article\_stub} is not null then timestamp\_diff(${event\_ts\_raw},${marketing\_content\_dim.interaction\_posted\_ts\_raw},DAY) end ;; } 2\. A table calculation that returns the running total (cumulative sum) of the measure you’re interested in, except in cases when that value is null in which case it also returns null; this is required in order to have each “ramp” curve terminate when there is no more data available for it ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F33803-11xwsrt_gd1i1vymhhytdgg.png) We can also use this data to calculate the ratio of **claps : visit value** delivered and **claps : visitor conversions** for each article, useful for us to understand for what category of topics positive feedback for an article translates to enquiries for our services. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F7377b-1rsvx0vvml95lu0fsb0xcdg.png) * * * Whilst our Squarespace site doesn’t comes with an API that lets us directly extract the names, URLs and posting dates for blogs we’ve cross-posted from Medium, once loaded into our BigQuery data warehouse we can derive that data by filtering Segment page events data to just those page events that have ‘/blog/2’ in the page URL, grouping the results and then taking the minimum timestamp value for those page events as the implied date the blog was posted. WITH pages AS ( SELECT page\_title, SPLIT(page\_url\_path,'/')\[safe\_OFFSET(5)\] AS article\_stub, MIN(event\_ts) posted\_at\_ts FROM {{ ref('stg\_segment\_events\_page\_events' ) }} WHERE page\_url\_path LIKE '%/blog/2%' AND page\_title != 'Rittman Analytics' GROUP BY 1, 2), first\_post AS ( SELECT \*, ROW\_NUMBER() OVER (PARTITION BY article\_stub) = 1 AS is\_first\_post FROM pages ), content as ( SELECT \* FROM first\_post WHERE is\_first\_post), If we then combine the standardised version of this data with the data we extracted earlier from Medium, our downstream warehouse tables now contain data on both the original and cross-posted versions of each article. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F807a0-1kn4mzodofx_njs5wlclj5a.png) Now I can calculate the **cumulative visit value** returned for just a single post but across both Medium and Squarespace as sources of [referrer traffic](https://wordpress.com/go/solve/what-is-a-referrer-how-does-it-help/); interestingly, whilst the Medium-hosted version of the post outperformed the Squarespace-hosted version for the first two months after publication, the Squarespace version continues to provide high-quality referral traffic. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F9e98d-1ks75gkillawftv0h7ihipg.png) The Medium-hosted version of that blog flatlined after a month and was overtaken by the Squarespace version a few weeks later; perhaps this was due to Medium ceasing to [recommend this post](https://buffer.com/library/how-to-use-medium/#:~:text=While%20there%20is%20no%20designated,generally%20been%20the%20most%20popular.) to readers who weren’t otherwise subscribers to our blog, a network effect we get benefit from when posting to a content aggregator site such as Medium.com. * * * If we now extend this data from just that captured from Medium and Squarespace to sources such as LinkedIn, Twitter and Github, we can get a full picture of the relative popularity of our blog articles and links to those articles published on each of these social networks. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2Fd85ef-1pazzolome80aj5mv1ncvka.png) If we also factor-in traffic that comes directly to us and via organic (Google, Bing-etc) searches, the latter two still contribute the vast majority of our conversions with direct traffic more likely to result in [a meeting being booked](https://calendly.com/markrittman/initial-discovery-call-with-mark-rittman) than traffic coming from either of our blog sites. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2Ffc64c-1_x-fg-yitykp6gzrdghehg.png) This is most likely because anyone visiting our site via typing it into their browser address bar is visiting our site for a business reason, and search traffic is similar in-reason although this does serve as a reminder for us to not forget SEO optimization when trying to generate more leads for our business. Taking that same set of data and charting the average visit value for each referral source we can see that visits originating from Medium (and LinkedIn) are above average in-terms of session visit value, and company-hosted blog page referral traffic is below-average in value. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F05823-1pwxcaseuiraqnrjwlvakzw.png) Finally though, visualizing the full set of metrics including session conversion rate and average value delivered we can in-fact see that our most valuable source of referral traffic is in-fact our [Github team site](https://github.com/rittmananalytics), where we’ve uploaded open-source dbt packages for the types of [data centralization tasks](https://github.com/rittmananalytics/ra_data_warehouse) we talk about on these blog articles as well as example dbt code for projects such as [multi-touch, multi-cycle marketing attribution](https://github.com/rittmananalytics/ra_attribution). ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F6ad5f-1up7gxjy_49ylwzr2xv_4va.png) ### Interested? Find out More Rittman Analytics is a boutique analytics consultancy specializing in the modern data stack who can help you [centralise your data sources](https://rittmananalytics.com/data-centralization) and [enable your end-users and data team](https://rittmananalytics.com/data-team-enablement) with best practices and a [modern analytics workflow](https://rittmananalytics.com/getting-started-with-dbt). Hopefully this look at how we try and measure the impact and value of our blogging activity has been of interest to you as a reader of this blog —our motivation when writing articles is of course really about a love for the topic and desire to share learnings but as data specialists, we also love measuring and analyzing our activity and using it to make that activity more effective and valuable. If you’re looking for some help and assistance understanding the impact and value delivered by your content or other marketing activity, or to help build-out your analytics capabilities and data team using a modern, flexible and modular data stack, [contact us now](https://calendly.com/markrittman/initial-discovery-call-with-mark-rittman?utm_source=medium.com&utm_medium=blog&utm_campaign=content_marketing_analytics) to organize a 100%-free, no-obligation call — we’d love to hear from you! --- # KPI Dashboards and Balanced Scorecards using Looker, dbt and Google BigQuery Canonical: https://rittmananalytics.com/blog/2022/07/19/kpi-dashboards-and-balanced-scorecards-using-looker-dbt-and-google-bigquery URL: https://rittmananalytics.com/blog/2022/07/19/kpi-dashboards-and-balanced-scorecards-using-looker-dbt-and-google-bigquery Description: One of the main use-cases for the data stack I described in How Rittman Analytics does Analytics Part 2 : Building our Modern Data Stack using dbt, Google BigQuery, Looker,… Author: Mark Rittman Published: 2022-07-19T14:14:00+01:00 Modified: 2022-07-19T14:14:00+01:00 --- One of the main use-cases for the data stack I described in [How Rittman Analytics does Analytics Part 2 : Building our Modern Data Stack using dbt, Google BigQuery, Looker, Segment and Rudderstack](https://blog.rittmananalytics.com/how-rittman-analytics-does-analytics-part-2-building-a-modern-data-stack-using-dbt-google-a0a17cdb4812) is to provide the data for our internal KPI dashboard shown below, built using dbt and Looker and powered by our Google BigQuery cloud data warehouse. In previous years we used just one KPI — revenue against target — when setting goals for the team but as our business has matured and we need to consider factors beyond just revenue if we’re to make that growth sustainable; this year we adopted a more holistic approach to thinking about performance management that aimed to consider the factors that lead to revenue growth rather than just the outcome itself. Based on the classic performance management technique [first outlined in Harvard Business Review](https://hbr.org/1992/01/the-balanced-scorecard-measures-that-drive-performance-2) back in 1992, we started by defining overall objectives for the four perspectives of Finance, Customer, Process and Innovation by which our performance was going to be measured: - to **increase revenue** for the **financial** perspective - to **increase customer satisfaction** for the **customer** perspective - to **decrease cost of delivery** for our **process** perspective - to **grow our capabilities** for the **innovation** perspective Each of these objectives would then have two initiatives by which we’d achieve those objectives, for example “Increase our certification level” for the “Grow our Capabilities” objective for the Innovation perspective. Some objectives for now would only have one initiative and others later on might have three or more, but one or two seemed a good starting point for now. Then, to calculate how well we were doing at the end of each month we’d take each objective’s set of initiatives and calculate the average performance to target score for the entire set, and then calculate our overall performance score from the objective scores by applying a weighting to them reflecting their relative importance to the business over each quarter. Gathering the data together for this dashboard required us combine data held across a number of the SaaS applications used to run our business; Hubspot for our CRM data, Xero for financial, Officevibe for staff surveys and Google Sheets for certification progress, for example. As you can see from the data source overlays over the dashboard screenshot shown below, most required at least two data sources for actual and target values and some, for example the KPI and scorecard tiles at the top of the page, requiring data from up to eight separate app and file sources. Which is where our [internal data stack](https://blog.rittmananalytics.com/how-rittman-analytics-does-analytics-part-2-building-a-modern-data-stack-using-dbt-google-a0a17cdb4812) comes in, or at least a subset of the stack concerned with loading, transforming and combining the data we need for this particular dashboard. Even then we still do some of that combining in the Looker reporting layer when, for example, we use the [merge results](https://www.google.com/search?q=looker+merge+results&rlz=1C5CHFA_enGB912GB912&oq=looker+merge+results&aqs=chrome.0.0i512l2j0i22i30l2j69i60l3.2047j0j4&sourceid=chrome&ie=UTF-8) feature to compare actual and [forecasted](https://blog.rittmananalytics.com/adding-forecasting-to-your-looker-report-and-dashboards-833670e569d9) new deals from one explore with the relevant target from another explore. With the design of the dashboard, the layering of sections is intentional; at the bottom of the page are Innovation measures that show progress against target for activities that then enable the delivery (Customer) measures, which in-turn drive improvement in operational (Process) measures and finally, our financial measures. At the top of the page are a set of metric tiles, one each for our most important KPIs, show last month’s performance compared to prior period. As we already have those metrics in the dashboard in the form of rolling 12-month line and bar charts, we created the metric tiles by duplicating the existing ones and limiting the rows returned to just the last complete month and month before that. Creating the scorecard tiles at the top of the page involved a bit more work; as you can’t base the calculation of one metric tile on the results of another, we had to painstakingly recreate one big merge results query to first calculate each underlying metric in-turn, then calculate the scorecard objective percentages-to-target and then finally, the weighted overall score. Alternatively, you could extract the SQL for each of the dashboard tile queries and use them as part of a [derived table LookML view](https://gist.github.com/markrittman/0c39b319ba5d416094b65d23f973bd3f) that you could add to your project, reference in a standalone explore and use as the data source for the dashboard tiles directly. Finally, so that we could store a history of changes for this dashboard’s design and if needed, revert back to an earlier design, we exported the dashboard design as LookML and saved it alongside the LookML views and models. The LookML version of the KPI dashboard then became the “canonical” version with the user dashboard, [stored in a regular Shared Folder](https://docs.looker.com/relnotes/v22-changelog#22.2) alongside the regular user dashboards, its design “fixed” but exportable back as a user dashboard should we wish to create a new version in the future. ## **Interested? Find out More** Rittman Analytics is a boutique analytics consultancy and Google Cloud Platform / Looker partner who can help you [centralise your data sources](https://rittmananalytics.com/data-centralization), modernize your analytics and [enable your end-users and data team](https://rittmananalytics.com/data-team-enablement) with a [modern BI workflow](https://rittmananalytics.com/getting-started-with-dbt). If you’re looking for some help and assistance building your own KPI dashboard in Looker or other BI tools, or to help build-out your analytics capabilities or data warehouse on a modern, flexible and modular data stack, [contact us now](https://calendly.com/markrittman/30min/?/?) to organize a 100%-free, no-obligation call — we’d love to hear from you! --- # How Rittman Analytics does Analytics Part 2 : Building a Modern Data Stack using dbt, Google… Canonical: https://rittmananalytics.com/blog/2022/05/30/how-rittman-analytics-does-analytics-part-2-building-a-modern-data-stack-using-dbt-google URL: https://rittmananalytics.com/blog/2022/05/30/how-rittman-analytics-does-analytics-part-2-building-a-modern-data-stack-using-dbt-google Description: A couple of years ago I wrote about how we’d started to use Looker, Stitch and Google BigQuery to start building out our own internal analytics capability, and around a… Author: Mark Rittman Published: 2022-05-30T09:31:10+01:00 Modified: 2026-07-02T00:30:13+01:00 --- ![How Rittman Analytics does Analytics Part 2 : Building a Modern Data Stack using dbt, Google…](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2Ff17db-113h8tfehanjbpb9dsxtpow.png) A couple of years ago I wrote about how we’d started to use Looker, Stitch and Google BigQuery to [start building out our own internal analytics capability](https://rittmananalytics.com/blog/2019/5/7/how-rittman-analytics), and around a year wrote about our use of dbt and our RA Data Warehouse framework to [centralize all of our SaaS data sources](https://rittmananalytics.com/blog/2020/5/28/introducing-the-ra-warehouse-dbt-framework-how-rittman-analytics-does-data-centralization) into Google BigQuery cloud data warehouse. Two years on and our operational data warehouse now centralizes the data from more than twenty SaaS applications, online services and other data sources that it then transforms, integrates and loads into a series of linked subject-area warehouses using [conformed dimensions](https://www.kimballgroup.com/data-warehouse-business-intelligence-resources/kimball-techniques/dimensional-modeling-techniques/conformed-dimension/). As our updated analytics platform architecture diagram shows below, we’re continuing to use Looker but also building-out a BI tool-independent metrics layer using dbt, and we’re also making use of reverse ETL tools from Hightouch and Rudderstack to enrich customer and client profiles in our CRM system, Hubspot. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2Ff17db-113h8tfehanjbpb9dsxtpow.png) In this blog I’ll walk through the four layers in our internal analytics platform architecture and talk about the approach we’ve taken, tools we’ve chosen and design patterns we’ve created as an example of how one business has gone about creating its own “modern data stack” using dbt, Google BigQuery, Looker and various other tools and an [Extract, Load and Transform](https://en.wikipedia.org/wiki/Extract,_load,_transform) approach to building and loading a data warehouse. ### Extract and Load (“EL”) Warehouse Layer Like most modern businesses we use a significant number of SaaS applications to run our business and use a mix of data pipeline services to centralize their data in our warehouse; the most straightforward are those that we can bring in using out-of-the-box connectors from our partners at Fivetran and Stitch, landing data in raw form into the warehouse. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2Fe64aa-1b-ynxxlttokhxdva2x98ew.png) For those services with REST APIs but no pre-built connectors we use [Coupler.io](https://www.coupler.io/) to connect to those APIs and load data from them in CSV format into BigQuery. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F9b484-1mvydxl3ip1ega04ypoo2aw.png) For apps that make their event stream consumable via webhooks or who have a [Zapier](https://zapier.com/) integration we capture those events either via webhook catches or by setting-up Zapier triggers, then package those events up to match [Segment’s HTTP API spec](https://segment.com/docs/connections/sources/catalog/libraries/server/http-api/) and then forward them on as a real-time event stream via Segment’s Connections customer data infrastucture, routing them to BigQuery as a warehouse destination as well as to other downstream customer and marketing services such as Intercom and Hubspot CRM. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F00505-1o3oyzcik7reccyhk55bmww.png) We also use a service called [Phantombuster](https://support.phantombuster.com/hc/en-001/articles/360015516339-What-is-a-Phantom-how-to-choose-one) to monitor and scrape social media sites such as LinkedIn and Twitter to capture interactions, comments and profile follows to build-up a profile of our wider audience and the type of topics that seem to drive engagement, feeding that data again using Coupler.io into the warehouse as a hosted CSV or JSON data source ### **Transform Layer** I’ve covered [our approach to data centralization and dbt](https://rittmananalytics.com/blog/2020/5/28/introducing-the-ra-warehouse-dbt-framework-how-rittman-analytics-does-data-centralization) on this blog a number of times before and [our design approach](https://blog.rittmananalytics.com/deduplicating-company-records-in-a-multi-source-data-centralization-project-using-dbt-google-b336a7ccfe61) is still holding up after integrating our twentieth data source a few weeks ago. We use dbt for modeling, transforming, validating and orchestrating our hourly data load and recently added a [metrics layer](https://docs.getdbt.com/docs/dbt-cloud/using-dbt-cloud/cloud-metrics-layer) to help us assess the value in including them as standard in our client delivery projects. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2Fcca7a-11heivbftjgdy-d8es2f9hg.png) Each data source contains a combination of data that’s unique to that source, for example journal entries in Xero or sales opportunities from Hubspot, together with other data elements that can be found in some or all of the other sources such as details on contacts, companies or projects we deliver. The dbt DAG below, for example, shows the various sources that combine to provide data for our companies dimension which in-turn is then referenced by almost every other fact table downstream in the data pipeline, allowing us to jump from sales activity to project delivery all the way to NPS scores and retention when analyzing our operational efficiency. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F0ed9d-10ffrkwli1ors8ykist_ulw.png) Our [dbt design pattern for entity resolution and deduplication](https://blog.rittmananalytics.com/deduplicating-company-records-in-a-multi-source-data-centralization-project-using-dbt-google-b336a7ccfe61) combines data from all of our sources of contact information, for example, with those sources enabled by a set of configuration entries in the package’s dbt\_project.yml YAML file: crm\_warehouse\_contact\_sources: \['hubspot\_crm','harvest\_projects','xero\_accounting','mailchimp\_email','calendly\_discovery','twitter\_followers','twitter\_likers','linkedin\_followers','github\_starrers','harvest\_forecast','linkedin\_commenters','linkedin\_likers','asana\_projects','jira\_projects','docusign\_contracts','hubspot\_nps','humaans\_hr'\] Each source of contact information is defined in a separate model within the staging area of our dbt package and named in a consistent way, such that the integration-layer dbt model downstream in the package can union-together those staging models using a Jinja loop and calls to the [dbt\_utils.union\_relations](https://github.com/dbt-labs/dbt-utils#union_relations-source) utility macro. {% if var('crm\_warehouse\_contact\_sources') %} {{config(materialized="table")}} with t\_merge\_list as ( {% set relations\_list = \[\] %} {% for source in var('crm\_warehouse\_contact\_sources') %} {% do relations\_list.append(ref('stg\_' ~ source ~ '\_contacts')) %} {% endfor %} {{ dbt\_utils.union\_relations( relations=relations\_list) }} ) ... select ... from t\_merge\_list {% else %} {{ config( enabled=false ) }} For sources that require deduplication and may require over-rides to be specified to merge records that otherwise wouldn’t merge on just name, we provide a seed file of IDs that need to merge along with files of targets and other reference data that we then include and load in each time the warehouse refresh runs. The second dbt project DAG shown below shows how data on contacts is sourced from virtually every data source we ingest and then used to link-together all of that contacts’ engagements with our business and channels of communication. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2Fc418e-16cqkdjwfzqrp5gmlcy7ndq.png) Using this dbt design pattern and the first-party data we collect gives us the ability to create complete 360-degree profiles of clients interests and engagements with our services and social media accounts. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2Fa3248-1hmjxotsorwtkyov_vj9p7w.png) ### Data Warehouse Data Models Data for analysis is then separated into a series of subject area warehouses that link-together through common dimensions such as contact, customer and project, with the overall warehouse data model its subject areas shown in the data model diagram below. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F42a20-1sm58m-p0xmpz4tupu3w1ja.png) Stepping through each subject warehouse in-turn, the **Sales and CRM** **Warehouse** is the most central of the subject area warehouses and contains customer and contact details together with calculated segments, channel and product preferences together with fact tables listing sales deals, communications and a complete event-level customer behavioral history ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F233d9-1ocrkioswmwv7m1ors9pg9a.png) Next, our **Finance and Invoicing** **Warehouse** sources from Xero, Harvest (our timesheet system) and Stripe and is used primarily for P&L reporting and calculating individual project profitability. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F0dc7c-1ixi57xprfdhl6k0k4xmoog.png) Next, our **Marketing Warehouse** stores all ad campaign and ad spend data along with content marketing and social media interactions, with [a separate warehouse and dbt package we’ve also created](https://github.com/rittmananalytics/ra_attribution) for ad spend and revenue attribution, campaign RoI and customer cost-of-acquisition. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2Feedc4-1zcidzv-qh9lko5cecfjvea.png) The other main subject area and the one that’s most consulting services-specific is our **Projects Warehouse**, sourcing data primarily from Harvest (our timesheets system) Jira and Docusign for contract activity, and is used for calculating our team’s utilization and resource forecasts. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F671c0-1heyyyhfubelgdvzcj5bowq.png) Other subject area warehouses include an **HR** **Warehouse** (with sensitive and personal data stores in a separate, restricted access GCP project) as well as a **Product Usage Warehouse** into which we stream log and usage data from our GCP projects, Looker and BigQuery accounts and log data from Fivetran. ### Activation and Analysis Layer The last but probably most important layer is the **Activation and Analysis Layer** where data is presented for analysis in the form of Looker dashboards, looks and explores; recently we’ve also been playing-around with an open-source Looker clone called [Lightdash](https://www.lightdash.com/) that [we covered on this blog a few weeks ago](https://rittmananalytics.com/blog/2022/2/1/lightdash-looker-and-dbt-as-the-bi-tool-metrics-layer) that uses the dbt metrics layer I mentioned earlier in this article as its business semantic model. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F4d64f-1d9zfe2e95zdt9lvqbpo4qa.png) For specific use-cases we use specialist consumption tools for tasks they’re better suited for, for example the notebook service [Hex](https://hex.tech/) that we used a couple of weeks ago to create the externally-facing data app I’ve embedded below, for example, to publish the listener stats for our podcast [Drill to Detail](https://rittmananalytics.com/drilltodetail). Finally and as we covered in our blogs on [Customer 360 Data Warehousing and Sync to Hubspot using BigQuery, dbt, Looker and Hightouch](https://rittmananalytics.com/blog/2021/2/15/customer-360-degree-analysis-and-hightouch) and [When and How Data Warehouses are the New Customer Data Platform](https://rittmananalytics.com/blog/2021/3/12/customer-data-warehouses-are-the-new-customer-data-platform), we sync the customer segments, content preferences and other calculated attributes from our warehouse back into our CRM and other marketing platforms using reverse ETL tools from Hightouch and Rudderstack, updating each system’s contact records using contact email as the common point of reference. ![](https://cdn-images-1.medium.com/max/800/0*J7U4CZ3BA6jXHrBU) ### Interested? Find out More Rittman Analytics is a boutique analytics consultancy specializing in the modern data stack who can help you [centralise your data sources](https://rittmananalytics.com/data-centralization) and [enable your end-users and data team](https://rittmananalytics.com/data-team-enablement) with best practices and a [modern analytics workflow](https://rittmananalytics.com/getting-started-with-dbt). As example code we’ve made the dbt package used to build our internal data warehouse available as a [public git repo](https://github.com/rittmananalytics/ra_warehouse_internal_clone) on our Github account and you’re free to check out how we’ve approached delivering our own data warehousing and data centralization project. And, of course, if you’re looking for some help and assistance building-out your analytics capabilities or data warehouse on a modern, flexible and modular data stack, [contact us now](https://calendly.com/markrittman/30min/?/?) to organize a 100%-free, no-obligation call — we’d love to hear from you and see how we might help you and your team build your own modern, ETL-style operational data warehouse. --- # Building our Modern Data Stack using dbt, Google BigQuery, Looker, Segment and Rudderstack — Rittman Analytics Canonical: https://rittmananalytics.com/blog/2022/05/30/building-our-modern-data-stack-using-dbt-google-bigquery-looker-segment-and-rudderstack-rittman-analytics URL: https://rittmananalytics.com/blog/2022/05/30/building-our-modern-data-stack-using-dbt-google-bigquery-looker-segment-and-rudderstack-rittman-analytics Description: How Rittman Analytics does Analytics Part 2 : Building our Modern Data Stack using dbt, Google BigQuery, Looker, Segment and Rudderstack. Modern Data Stack 30 May Written By Mark Rittman… Author: Mark Rittman Published: 2022-05-30T01:00:00+01:00 Modified: 2022-05-30T01:00:00+01:00 --- ![Building our Modern Data Stack using dbt, Google BigQuery, Looker, Segment and Rudderstack — Rittman Analytics](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2022%2F05%2F1752872279158-8rgdhf.png%3Ffit%3D1400%252C620%26ssl%3D1) # How Rittman Analytics does Analytics Part 2 : Building our Modern Data Stack using dbt, Google BigQuery, Looker, Segment and Rudderstack. [Modern Data Stack](https://rittmananalytics.com/blog/category/Modern+Data+Stack) 30 May Written By [Mark Rittman](https://rittmananalytics.com/blog?author=63c478ac6811232b3153d585) A couple of years ago I wrote about how we’d started to use Looker, Stitch and Google BigQuery to [start building out our own internal analytics capability](https://rittmananalytics.com/blog/2019/5/7/how-rittman-analytics), and around a year wrote about our use of dbt and our RA Data Warehouse framework to [centralise all of our SaaS data sources](https://rittmananalytics.com/blog/2020/5/28/introducing-the-ra-warehouse-dbt-framework-how-rittman-analytics-does-data-centralization) into Google BigQuery cloud data warehouse. Two years on and our operational data warehouse now centralises the data from more than twenty SaaS applications, online services and other data sources that it then transforms, integrates and loads into a series of linked subject-area warehouses using [conformed dimensions](https://www.kimballgroup.com/data-warehouse-business-intelligence-resources/kimball-techniques/dimensional-modeling-techniques/conformed-dimension/). As our updated analytics platform architecture diagram shows below, we’re continuing to use Looker but also building-out a BI tool-independent metrics layer using dbt, and we’re also making use of reverse ETL tools from Hightouch and Rudderstack to enrich customer and client profiles in our CRM system, Hubspot. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872279158-8rgdhf.png%3Fresize%3D1400%252C620%26ssl%3D1) In this blog I’ll walk through the four layers in our internal analytics platform architecture and talk about the approach we’ve taken, tools we’ve chosen and design patterns we’ve created as an example of how one business has gone about creating its own “modern data stack” using dbt, Google BigQuery, Looker and various other tools and an [Extract, Load and Transform](https://en.wikipedia.org/wiki/Extract,_load,_transform) approach to building and loading a data warehouse. ## Extract and Load (“EL”) Warehouse Layer Like most modern businesses we use a significant number of SaaS applications to run our business and use a mix of data pipeline services to centralize their data in our warehouse; the most straightforward are those that we can bring in using out-of-the-box connectors from our partners at Fivetran and Stitch, landing data in raw form into the warehouse. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872279392-jthxn.png%3Fresize%3D702%252C750%26ssl%3D1) For those services with REST APIs but no pre-built connectors we use [Coupler.io](https://www.coupler.io/) to connect to those APIs and load data from them in CSV format into BigQuery. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872279366-nh4zji.png%3Fresize%3D1400%252C968%26ssl%3D1) For apps that make their event stream consumable via webhooks or who have a [Zapier](https://zapier.com/) integration we capture those events either via webhook catches or by setting-up Zapier triggers, then package those events up to match [Segment’s HTTP API spec](https://segment.com/docs/connections/sources/catalog/libraries/server/http-api/) and then forward them on as a real-time event stream via Segment’s Connections customer data infrastucture, routing them to BigQuery as a warehouse destination as well as to other downstream customer and marketing services such as Intercom and Hubspot CRM. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872279570-lqhf5c.png%3Fresize%3D1390%252C1066%26ssl%3D1) We also use a service called [Phantombuster](https://support.phantombuster.com/hc/en-001/articles/360015516339-What-is-a-Phantom-how-to-choose-one) to monitor and scrape social media sites such as LinkedIn and Twitter to capture interactions, comments and profile follows to build-up a profile of our wider audience and the type of topics that seem to drive engagement, feeding that data again using Coupler.io into the warehouse as a hosted CSV or JSON data source ## Transform (“T”) Layer I’ve covered [our approach to data centralization and dbt](https://rittmananalytics.com/blog/2020/5/28/introducing-the-ra-warehouse-dbt-framework-how-rittman-analytics-does-data-centralization) on this blog a number of times before and [our design approach](https://blog.rittmananalytics.com/deduplicating-company-records-in-a-multi-source-data-centralization-project-using-dbt-google-b336a7ccfe61) is still holding up after integrating our twentieth data source a few weeks ago. We use dbt for modeling, transforming, validating and orchestrating our hourly data load and recently added a [metrics layer](https://docs.getdbt.com/docs/dbt-cloud/using-dbt-cloud/cloud-metrics-layer) to help us assess the value in including them as standard in our client delivery projects. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872279230-2k30mc.png%3Fresize%3D1206%252C992%26ssl%3D1) Each data source contains a combination of data that’s unique to that source, for example journal entries in Xero or sales opportunities from Hubspot, together with other data elements that can be found in some or all of the other sources such as details on contacts, companies or projects we deliver. The dbt DAG below, for example, shows the various sources that combine to provide data for our companies dimension which in-turn is then referenced by almost every other fact table downstream in the data pipeline, allowing us to jump from sales activity to project delivery all the way to NPS scores and retention when analyzing our operational efficiency. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872279394-1vlkxu.png%3Fresize%3D1400%252C476%26ssl%3D1) Our [dbt design pattern for entity resolution and deduplication](https://blog.rittmananalytics.com/deduplicating-company-records-in-a-multi-source-data-centralization-project-using-dbt-google-b336a7ccfe61) combines data from all of our sources of contact information, for example, with those sources enabled by a set of configuration entries in the package’s dbt\_project.yml YAML file: crm\_warehouse\_contact\_sources: \[‘hubspot\_crm’,’harvest\_projects’,’xero\_accounting’,’mailchimp\_email’,’calendly\_discovery’,’twitter\_followers’,’twitter\_likers’,’linkedin\_followers’,’github\_starrers’,’harvest\_forecast’,’linkedin\_commenters’,’linkedin\_likers’,’asana\_projects’,’jira\_projects’,’docusign\_contracts’,’hubspot\_nps’,’humaans\_hr’\] Each source of contact information is defined in a separate model within the staging area of our dbt package and named in a consistent way, such that the integration-layer dbt model downstream in the package can union-together those staging models using a Jinja loop and calls to the [dbt\_utils.union\_relations](https://github.com/dbt-labs/dbt-utils#union_relations-source) utility macro. {% if var(‘crm\_warehouse\_contact\_sources’) %}{{config(materialized=”table”)}}with t\_merge\_list as({% set relations\_list = \[\] %}{% for source in var(‘crm\_warehouse\_contact\_sources’) %}{% do relations\_list.append(ref(‘stg\_’ ~ source ~ ‘\_contacts’)) %}{% endfor %}{{ dbt\_utils.union\_relations(relations=relations\_list)}})…select…fromt\_merge\_list{% else %}{{config(enabled=false)}} For sources that require deduplication and may require over-rides to be specified to merge records that otherwise wouldn’t merge on just name, we provide a seed file of IDs that need to merge along with files of targets and other reference data that we then include and load in each time the warehouse refresh runs. The second dbt project DAG shown below shows how data on contacts is sourced from virtually every data source we ingest and then used to link-together all of that contacts’ engagements with our business and channels of communication. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872279514-3js6je.png%3Fresize%3D1400%252C645%26ssl%3D1) Using this dbt design pattern and the first-party data we collect gives us the ability to create complete 360-degree profiles of clients interests and engagements with our services and social media accounts. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872279478-y3cs7r.png%3Fresize%3D1272%252C1078%26ssl%3D1) ## Data Warehouse Data Models Data for analysis is then separated into a series of subject area warehouses that link-together through common dimensions such as contact, customer and project, with the overall warehouse data model its subject areas shown in the data model diagram below. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872279480-ba322d.png%3Fresize%3D1400%252C1098%26ssl%3D1) Stepping through each subject warehouse in-turn, the **Sales and CRMWarehouse** is the most central of the subject area warehouses and contains customer and contact details together with calculated segments, channel and product preferences together with fact tables listing sales deals, communications and a complete event-level customer behavioral history ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872279524-u2vmtg.png%3Fresize%3D1400%252C1109%26ssl%3D1) Next, our **Finance and InvoicingWarehouse** sources from Xero, Harvest (our timesheet system) and Stripe and is used primarily for P&L reporting and calculating individual project profitability. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872279476-1pjt5c.png%3Fresize%3D1400%252C1049%26ssl%3D1) Next, our **Marketing Warehouse** stores all ad campaign and ad spend data along with content marketing and social media interactions, with [a separate warehouse and dbt package we’ve also created](https://github.com/rittmananalytics/ra_attribution) for ad spend and revenue attribution, campaign RoI and customer cost-of-acquisition. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872279445-wfpuuj.png%3Fresize%3D1400%252C1601%26ssl%3D1) The other main subject area and the one that’s most consulting services-specific is our **Projects Warehouse**, sourcing data primarily from Harvest (our timesheets system) Jira and Docusign for contract activity, and is used for calculating our team’s utilization and resource forecasts. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872279695-5w9kpf.png%3Fresize%3D1400%252C1239%26ssl%3D1) Other subject area warehouses include an **HRWarehouse** (with sensitive and personal data stores in a separate, restricted access GCP project) as well as a **Product Usage Warehouse** into which we stream log and usage data from our GCP projects, Looker and BigQuery accounts and log data from Fivetran. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872279489-yx843b.png%3Fresize%3D1400%252C1239%26ssl%3D1) ## Activation and Analysis Layer The last but probably most important layer is the **Activation and Analysis Layer** where data is presented for analysis in the form of Looker dashboards, looks and explores; recently we’ve also been playing-around with an open-source Looker clone called [Lightdash](https://www.lightdash.com/) that [we covered on this blog a few weeks ago](https://rittmananalytics.com/blog/2022/2/1/lightdash-looker-and-dbt-as-the-bi-tool-metrics-layer) that uses the dbt metrics layer I mentioned earlier in this article as its business semantic model. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872279461-441fic.png%3Fresize%3D1400%252C1052%26ssl%3D1) For specific use-cases we use specialist consumption tools for tasks they’re better suited for, for example the notebook service [Hex](https://hex.tech/) that we used a couple of weeks ago to create the externally-facing data app I’ve embedded below, for example, to publish the listener stats for our podcast [Drill to Detail](https://rittmananalytics.com/drilltodetail). Finally and as we covered in our blogs on [Customer 360 Data Warehousing and Sync to Hubspot using BigQuery, dbt, Looker and Hightouch](https://rittmananalytics.com/blog/2021/2/15/customer-360-degree-analysis-and-hightouch) and[When and How Data Warehouses are the New Customer Data Platform](https://rittmananalytics.com/blog/2021/3/12/customer-data-warehouses-are-the-new-customer-data-platform), we sync the customer segments, content preferences and other calculated attributes from our warehouse back into our CRM and other marketing platforms using reverse ETL tools from Hightouch and Rudderstack, updating each system’s contact records using contact email as the common point of reference. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872279525-ix5p3k.png%3Fresize%3D1400%252C820%26ssl%3D1) ## Interested? Find out More Rittman Analytics is a boutique analytics consultancy specialising in the modern data stack who can help you [centralise your data sources](https://rittmananalytics.com/data-centralization) and [enable your end-users and data team](https://rittmananalytics.com/data-team-enablement) with best practices and a [modern analytics workflow](https://rittmananalytics.com/getting-started-with-dbt). As example code we’ve made the dbt package used to build our internal data warehouse available as a [public git repo](https://github.com/rittmananalytics/ra_warehouse_internal_clone) on our Github account and you’re free to check out how we’ve approached delivering our own data warehousing and data centralization project. And, of course, if you’re looking for some help and assistance building-out your analytics capabilities or data warehouse on a modern, flexible and modular data stack, [contact us now](https://calendly.com/markrittman/30min/?/?) to organize a 100%-free, no-obligation call — we’d love to hear from you and see how we might help you and your team build your own modern, ETL-style operational data warehouse. [Analytics](https://rittmananalytics.com/blog/tag/Analytics)[Tips & Techniques](https://rittmananalytics.com/blog/tag/Tips+%26+Techniques) [Mark Rittman](https://rittmananalytics.com/blog?author=63c478ac6811232b3153d585) --- # Stitching Identity Across the Customer Journey — Rittman Analytics Canonical: https://rittmananalytics.com/blog/2022/05/30/stitching-identity-across-the-customer-journey-rittman-analytics URL: https://rittmananalytics.com/blog/2022/05/30/stitching-identity-across-the-customer-journey-rittman-analytics Description: Stitching Identity Across the Customer Journey Using Segment, Google BigQuery and Looker Marketing Analytics 30 May Written By Mark Rittman Last week in my previous blog “How Rittman Analytics does… Author: Mark Rittman Published: 2022-05-30T01:00:00+01:00 Modified: 2022-05-30T01:00:00+01:00 --- ![Stitching Identity Across the Customer Journey — Rittman Analytics](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2022%2F05%2F1752872263896-82utpb.png%3Ffit%3D1400%252C808%26ssl%3D1) # Stitching Identity Across the Customer Journey Using Segment, Google BigQuery and Looker [Marketing Analytics](https://rittmananalytics.com/blog/category/Marketing+Analytics) 30 May Written By [Mark Rittman](https://rittmananalytics.com/blog?author=5846da08f5e231d83261e221) Last week in my previous blog [“How Rittman Analytics does Analytics Part 2 : Building our Modern Data Stack using dbt, Google BigQuery, Looker, Segment and Rudderstack”](https://blog.rittmananalytics.com/how-rittman-analytics-does-analytics-part-2-building-a-modern-data-stack-using-dbt-google-a0a17cdb4812) I went through how we’ve built our own modern data stack and use it to record and analyze client interactions from initial sales enquiry through to invoicing and renewal, by creating a unique record for each of our clients and using it to link together all of their activity in Xero, Hubspot, Jira and our other SaaS applications. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872263896-82utpb.png%3Fresize%3D1400%252C808%26ssl%3D1) By centralizing all of this client activity in a cloud data warehouse one of the most valuable use-cases we have for this data is to help us understand our customer journey, from initial awareness through to service and (if all goes well) loyalty. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872264028-pc3y96.png%3Fresize%3D1400%252C515%26ssl%3D1) Data from Hubspot, Xero and our other SaaS applications helps us connect our customer touchpoints at the acquisition, service and loyalty stages but what about the ones before acquisition, when prospects come to our site anonymously or visit our profiles on LinkedIn or Twitter? When they only reveal who they are by booking a meeting on Calendly, how do we link those identities to the anonymous visits they’ve made to our website? ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872263905-hjh3dn.png%3Fresize%3D1400%252C808%26ssl%3D1) In this blog I’ll explain how we connect prospect and client activity together across all of the stages in our customer journey, and how we try and extend this customer journey to platforms we don’t own or control but do provide some limited information on who’s interested in our content and services. I’ll use Segment for the examples but we do something similar with tracking data we get from Rudderstack and Snowplow which use similar concepts around tracking and identifying users and their behavior. ## Tracking Anonymous Activity on your Site Starting with the simplest example of a visitor arriving at our website and browsing a series of pages, all of that activity would be anonymous but connected via their [*anonymousId*](https://segment.com/docs/connections/sources/catalog/libraries/website/javascript/identity/), a randomly generated 36 character string automatically assigned to the user’s browser or device on first visit. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872264069-qb28m.png%3Fresize%3D1400%252C969%26ssl%3D1) This cookie-stored first-party visitor ID will persist between sessions for up to twelve months but expires after seven days of inactivity when stored in browsers based on [Apple’s WebKit engine](https://webkit.org/blog/10218/full-third-party-cookie-blocking-and-more/), for example Safari on MacOS or all browsers on running on iOS. The impact of this eventual expiry is that an individual user visiting your site may have multiple IP addresses associated with a single anonymousId, and second or multiple other anonymousIds linked to different devices or the same device used after an extended period of inactivity. We can see this in our Segment event data stored in our Google BigQuery data warehouse by querying it and then grouping by anonymousId (note that the SQL expression around the context\_ip column was added to obfuscate IP addresses in the example output here and later in this blog) selectanonymous\_id,array\_agg(struct(timestamp,concat(split(context\_ip,’.’)\[safe\_offset(0)\],’.\*\*\*.\*\*\*.’,split(context\_ip,’.’)\[safe\_offset(3)\]) as context\_ip,context\_page\_path,context\_page\_referrer,context\_page\_title,context\_user\_agent)order by timestamp) page\_viewfrom\`ra-development.company\_website.pages\`group by1order by1 See how the for the first anonymousId the IP address is constant, whereas for the second their IP address has changed over time; in addition, both of these anonymousIds may in-fact be linked to the same person, you just wouldn’t know it from the data as its currently shown. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872264234-st3l2.png%3Fresize%3D1400%252C441%26ssl%3D1) The way we link two or more anonymousIds and their activity together is by your website or app sending an [Identify](https://segment.com/docs/connections/spec/identify/) event, ideally each time a user logs in and identifies themselves or provides you with useful information such as channel or product preferences that you want stored in their user profile. However … like most consulting websites ours doesn’t allow visitors to create accounts or log in, something that’s typical of sites that either aren’t transactional or are but don’t want to introduce unnecessary friction to their checkout process. ## Linking Offsite Conversions to Onsite Anonymous Activity We don’t have a checkout on our site but we do have what we consider as our conversion event; when a visitor books a meeting with us using our Calendly service, either by pressing one of the Contact Us buttons on various pages on the site or by filling in the contact form on our home page. Both conversion methods collect the visitor’s name, email address and reason for enquiring and, short of having a user login and ability to register for an account, are the best we’re going to get for our site in-terms of users identifying themselves. Whilst [Segment’s best practices](https://segment.com/docs/connections/spec/best-practices-identify/) advises against using email as UserId as it can vary over time for someone, we use it as ID at prospect stage and then consolidate all emails and source system IDs into single contact records once they appear in Hubspot, our CRM system. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872264115-hywd0p.png%3Fresize%3D1400%252C277%26ssl%3D1) Calendly doesn’t directly integrate with Segment but does have an integration with Zapier, which means we can set-up a routine or “Zap” that triggers when a Calendly event is booked and then sends two events to Segment via its HTTP API; a track event to record the booking being made followed by an identify event that we use to link these anonymous visitor activities to an email address and prospect name. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872264118-hmpjql.png%3Fresize%3D1400%252C900%26ssl%3D1) Normally a SaaS application event captured by Zapier and sent to Segment would not be aware of the visitor website session that originated it, as it would be Calendly that triggered the routine to run as far as Zapier was concerned. However it is possible to pass information via UTM parameters we add to the e Calendly API calls we attach to the Contact Us buttons on our website, for example using the code snippet below: var links = document.getElementsByClassName(“sqs-block-button-element”);for (i = 0; i < links.length; i++) {links\[i\].addEventListener(‘click’, function() {Calendly.initPopupWidget({url:’[https://calendly.com/markrittman/initial-discovery-call-with-mark-rittman?utm\_source=rittmananalytics.com&utm=medium=website&utm\_campaign=homepage\_hero\_image&utm\_term=’+analytics.user().anonym](https://calendly.com/markrittman/initial-discovery-call-with-mark-rittman?utm_source=rittmananalytics.com&utm=medium=website&utm_campaign=homepage_hero_image&utm_term=%E2%80%99+analytics.user\(\).anonym)ousId()});return false;});}; We use this ability to pass the visitor anonymousId to Calendly via the otherwise unusedutm\_term parameter, which the Calendly API then makes available back to us in Zapier and we use in the track and identify events we send to Segment. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872263897-w6787q.png%3Fresize%3D1400%252C1051%26ssl%3D1) There’s one other source of identification I can make use of at this Consideration stage of our customer journey; the email address we embed in each link URL and we can extract from the page URL querystring in any page view event that is the result of one of those links being clicked. coalesce(split(split(context\_page\_url,’ajs\_aid=’)\[safe\_offset(1)\],’&’)\[safe\_offset(0)\],user\_id) as user\_id, Then, if the visitor returns at a later date with a different AnonymousId and books a second meeting using the same email address, both sets of events and anonymousIds in the downstream warehouse can be linked via their common UserId. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872264120-fmwfbe.png%3Fresize%3D1400%252C1077%26ssl%3D1) The way you do this linking of anonymousIds their common UserIds is by using ID-stitching logic such as the excerpt from [this full SQL query](https://gist.github.com/markrittman/420b8b571f62fb62230eea10c0fe5ff5) below: …id\_stitching as (selectdistinct anonymous\_id as anonymous\_id,last\_value(user\_id ignore nulls) over (partition by anonymous\_id order by timestamp rows between unbounded preceding and unbounded following ) as user\_id,min(timestamp) over (partition by anonymous\_id ) as first\_seen\_at,max(timestamp) over (partition by anonymous\_id ) as last\_seen\_atfromevents ),mapped as (selectcoalesce(i.user\_id,e.anonymous\_id) as blended\_user\_id,e.\*fromevents eleft joinid\_stitching iusing(anonymous\_id)),names\_backfilled as (select\* except (name,email),last\_value(email ignore nulls) over (partition by blended\_user\_id order by timestamp rows between unbounded preceding and unbounded following ) as email,last\_value(name ignore nulls) over (partition by blended\_user\_id order by timestamp rows between unbounded preceding and unbounded following ) as namefrommapped ),… If you then use the name and email address you’ve now obtained for this UserId and use it to back-fill all of the events that only had anonymousIds, you can then group each users’s events by those details and output those as a more complete record of each user’s behavior at the Consideration stage of the user journey. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872264132-quyhbh.png%3Fresize%3D1400%252C311%26ssl%3D1) We can use this data to understand, for example. the most common user journeys that are a result of visitors landing on some of our blog posts, with the first step in this analysis being to pivot each users’ page views into a series of columns for first page viewed, second page viewed and so on. If we then paste this query in Looker’s SQL Runner and visualize the results using the Sankey data vizualization plug-in available on the Looker Marketplace, I can see that our article on Multi-Channel Attribution had by far the most initial visitors and widest range of subsequent page views. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872264104-xmo0wf.png%3Fresize%3D1400%252C574%26ssl%3D1) Or I can visualize this visitor journey as a Looker table visualization, pivoting the event sequence number and setting page title as a measure ([SQL here](https://gist.github.com/markrittman/19cddf0884acc5acc4b413a9f5a7d599)). ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872264100-opubnm.png%3Fresize%3D1400%252C366%26ssl%3D1) ## Tracking Offsite Activity So far we’ve just looked at the Consideration stage in our customer journey, but what about the Awareness stage? Taking our business as an example, our client journeys often start by someone first becoming aware of us through an article posted on Twitter, LinkedIn or Medium and then some time later, following a link to our site and finding out about our services. The common factor in all these Awareness-stage interactions is that they happen on platforms that we don’t control; we can’t add Segment tracking code to our profile page on LinkedIn, for example, and by default can only make use of UTM parameters in article URLs to give us a general sense of where site traffic is coming from. What is also possible though using scraping services such as PhantomBuster is to collect some limited details of interactions with our content on sites like LinkedIn, providing us with the names and in some cases email addresses of users interacting with our content and the nature of those interactions, for example liking a post or following our profile. Using the user’s name and potentially their email address it’s possible to then infer that they are the same people who’ve visited our site, once they later identify themselves by booking a meeting, by matching those social media users to our known users by name. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872264120-lh4ynd.png%3Fresize%3D1400%252C854%26ssl%3D1) What’s important to remember with this type of identity stitching is that, unlike positive link we make between activities on our website and Calendly bookings by passing across the visitor’s AnonymousID, in this instance we’re inferring the link by connecting people by just their name. As such, we add a boolean flag to each event record to indicate whether the visitor identity is inferred or not, as shown in the excerpt below from the full and final SQL query [here](https://gist.github.com/markrittman/d14a40a0b4e44cc89379100955aa3ae8). selectblended\_user\_id,name,email,logical\_or(is\_inferred\_identity) as is\_inferred\_identity,array\_agg(struct(timestamp,event\_type,context\_ip,anonymous\_id,user\_id,is\_inferred\_identity,context\_page\_path,context\_page\_referrer,context\_page\_title,context\_user\_agent,context\_campaign\_source,context\_campaign\_medium,context\_campaign\_source,requirement)order by timestamp) eventfrom names\_backfilledgroup by 1,2,3 The output from this final query allows us to finally stitch together the off-platform Awareness stage of our customer journey with the anonymous and subsequently-identified Consideration stage activity of visitors to our website, as shown for example for myself in the final screenshot below. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872264094-fuqd8b.png%3Fresize%3D1400%252C365%26ssl%3D1) ## Interested? Find out More Rittman Analytics is a boutique analytics consultancy specializing in the modern data stack who can help you [centralise your data sources](https://rittmananalytics.com/data-centralization) and [enable your end-users and data team](https://rittmananalytics.com/data-team-enablement) with best practices and a [modern analytics workflow](https://rittmananalytics.com/getting-started-with-dbt). If you’re looking for some help and assistance understanding and analysing your customer journey, or to help build-out your analytics capabilities or data warehouse on a modern, flexible and modular data stack, [contact us now](https://calendly.com/markrittman/30min/?/?) to organize a 100%-free, no-obligation call — we’d love to hear from you and see how we might help you and your team build your own modern, ELT-style modern data stack. [Identity](https://rittmananalytics.com/blog/tag/Identity)[Looker](https://rittmananalytics.com/blog/tag/Looker)[Segment](https://rittmananalytics.com/blog/tag/Segment)[Marketing Attribution](https://rittmananalytics.com/blog/tag/Marketing+Attribution) [Mark Rittman](https://rittmananalytics.com/blog?author=5846da08f5e231d83261e221) CEO of Rittman Analytics, host of the Drill to Detail Podcast, ex-product manager and twice company founder. [https://rittmananalytics.com](https://rittmananalytics.com/) --- # Presenting on dbt & Firebolt at the Budapest dbt Meetup, Tuesday 10th May 2022 Canonical: https://rittmananalytics.com/blog/2022/05/05/presenting-on-dbt-firebolt-at-the-budapest-dbt-meetup-tuesday-10th-may-2022 URL: https://rittmananalytics.com/blog/2022/05/05/presenting-on-dbt-firebolt-at-the-budapest-dbt-meetup-tuesday-10th-may-2022 Description: If you found my blog post on using Firebolt Data Warehouse and Looker to analyze the Hacker News public dataset, I’m presenting a follow-on session at the Budapest dbt Meetup… Author: Mark Rittman Published: 2022-05-05T22:22:35+01:00 Modified: 2022-05-05T22:22:35+01:00 --- If you found my blog post on using [Firebolt Data Warehouse and Looker to analyze the Hacker News public dataset](https://rittmananalytics.com/blog/2022/4/25/analyzing-the-hacker-news-public-dataset-using-firebolt-data-warehouse-and-looker), I’m presenting a follow-on session at the [Budapest dbt Meetup](https://www.meetup.com/budapest-dbt-meetup/events/284955175/) next Tuesday on using dbt with Firebolt data warehouse. I’m co-presenting with Son N. Nguyen from another one of our partner, Budapest-based [HiFly Labs](https://hiflylabs.hu/english.html), as Sonny and I have been working closely together on an implementation of Firebolt, dbt and Looker for one of our joint clients. I’ll be presenting remotely with Sonny there in-person, both of us answering questions during and after the presentation. If you’re interested in finding out how Firebolt data warehouse works and how the new Firebolt dbt adapter helps you load, transform and index huge volumes of high-value analytics datasets, either [come along to the meetup](https://www.meetup.com/budapest-dbt-meetup/events/284955175/) if you’re in the area or [connect remotely and watch the webcast](https://bit.ly/3iO6Nvu). I’m looking forward to it! --- # Analyzing the Hacker News Public Dataset using Firebolt Data Warehouse and Looker Canonical: https://rittmananalytics.com/blog/2022/04/25/analyzing-the-hacker-news-public-dataset-using-firebolt-data-warehouse-and-looker URL: https://rittmananalytics.com/blog/2022/04/25/analyzing-the-hacker-news-public-dataset-using-firebolt-data-warehouse-and-looker Description: Something I’ve been meaning to write about for a while now is Firebolt, a relatively new startup who’ve been busy raising funds and hiring superstars with the aim of building… Author: Mark Rittman Published: 2022-04-25T07:24:58+01:00 Modified: 2026-07-02T00:30:07+01:00 --- ![Analyzing the Hacker News Public Dataset using Firebolt Data Warehouse and Looker](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F521fd-1e1pumyrs37q_j2wwfppr7w.png) Something I’ve been meaning to write about for a while now is [Firebolt](https://www.firebolt.io/?utm_source=partner&utm_medium=referral&utm_campaign=rittman_analytics), a relatively new startup who’ve been [busy raising funds](https://saasindustry.com/news/firebolt-raises-1-4-billion-in-series-c-funding-to-build-the-worlds-fastest-and-most-versatile-cloud-data-warehouse/) and [hiring superstars](https://www.crn.com/news/applications-os/firebolt-raises-100m-in-funding-round-hires-google-exec-as-cto) with the aim of building a cloud data warehouse that’s faster, more scalable and cheaper to run than Snowflake data warehouse. We’ve been working with Firebolt’s technology and implementation team over the past six months implementing Firebolt Data Warehouse for one of our clients and whilst for confidentiality reasons I can’t talk about that particular project, it has given me a fairly detailed understanding of how Firebolt works and what types of use-case and query workload makes sense to use it for. In full disclosure we’re now a Firebolt Partner but we’re also Snowflake, Google Cloud Platform and Oracle Partners, as well as regular uses of AWS Redshift, AWS Athena, Clickhouse, Postgres and many other database technologies; Firebolt is an interesting technology that aims to be both faster and cheaper than Snowflake and Google BigQuery, so how does it work and how does it stack-up against its more established rivals? * * * Firebolt is a new data warehousing technology that promises sub-second response times on very large datasets with highly-concurrent workloads — think of it as “Snowflake with Indexes”, or more succinctly — “speed of Clickhouse meets Snowflake architecture”. Firebolt’s underlying technology platform [has its roots in Clickhouse](https://www.youtube.com/watch?v=9rW9uEJ15tU), an open-source high-performance OLAP database system originally developed by Yandex, now Clickhouse, Inc to generate analytical reports in real-time from non-aggregated data that‘s constantly added-to in real-time. Clickhouse’s vectorized column store technology could scale-up and run distributed queries on multiple nodes but required the user to setup and manage the infrastructure it all ran on; Firebolt hard-forked the open-source Clickhouse code and [re-engineered it to become server-less](https://www.youtube.com/watch?v=9rW9uEJ15tU), decoupling storage and compute and added a metadata layer, query planning and service and orchestration layers as shown in the diagram below from [Firebolt’s technology whitepaper](https://www.firebolt.io/resources/firebolt-cloud-data-warehouse-whitepaper/?utm_source=partner&utm_medium=referral&utm_campaign=rittman_analytics). ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F521fd-1e1pumyrs37q_j2wwfppr7w.png) The thing that makes Firebolt fast though is the choices they’ve made around storage technology; cloud data warehouses such as Snowflake and BigQuery used made it possible to store and query effectively limitless amounts of structured and semi-structured data but did so using technologies available at the time that provided scale and flexibility but response times limited by network bandwidth and latency. Firebolt’s F3 (“Firebolt File Format”) reintroduces our old friend the “sparse index” — a concept very familiar to readers of this blog familiar with multi-dimensional OLAP technologies — to this story, making it possible to access data in the storage layer in a much more granular way than with partitions or micro-partitions, enabling more processing in-memory and less moving of data across slow networks and therefore — in theory, at least — providing lower-latency query response times for end-users. So how does Firebolt work in-practice? Let’s start by taking the [Hacker News public dataset](https://cloud.google.com/bigquery/public-data) kindly hosted by Google that contains all stories and comments since it launched back in 2006, load it into Firebolt and see how it all works. * * * The Hacker News public dataset comes in the form of three tables, “stories”, “comments” and “full” with the first two tables being just stories and posts over a limited period of time and the latter containing jobs and other postings for a total of 31m rows and 11GB of storage consumed. I’ve already exported the contents of those tables into an AWS S3 bucket as uncompressed parquet files, one directory per table exported, ready for loading into Firebolt. Very similar to staging and loading data into Snowflake, you first create an external table that maps onto those files sitting in our S3 bucket, like this: CREATE EXTERNAL TABLE HN\_FULL ( title STRING , url STRING, "text" STRING, dead BOOLEAN, "by" STRING, score INT, "time" INT, "timestamp" TIMESTAMP , type STRING, id INT, parent INT, descendants INT, ranking INT, deleted BOOLEAN ) URL = 's3://ra-hacker-news/full/' CREDENTIALS = ( AWS\_KEY\_ID = '' AWS\_SECRET\_KEY = '') OBJECT\_PATTERN = '\*.parquet' TYPE = ( PARQUET ); Then you create the table into which you’ll load this externally-staged “raw” data, with this table then storing data in Firebolt’s F3 storage format optimized for low-latency, super-fast queries. CREATE FACT TABLE HACKER\_NEWS\_ALL ( title STRING , url STRING, "text" STRING, dead BOOLEAN, author STRING, score INT, "time" INT, "timestamp" TIMESTAMP , month TIMESTAMP, type STRING, id INT, parent INT, descendants INT, ranking INT, deleted BOOLEAN ) PRIMARY INDEX type, month, author; Note two things about this DDL statement; the addition of “FACT” into the CREATE TABLE command and the PRIMARY INDEX clause at the end. Firebolt has three types of table you can define: - **External tables** that map onto external data stages and use whatever storage format (parquet, CSV, TSV, JSON) those files are stored in - **Fact tables** that are sharded across engine nodes, with each node storing part of the table. You use fact tables for your larger and most granular (transaction) tables, such as the full Hacker News dataset table export we’re working with here - **Dimension tables** are **replicated** in each engine node and are designed for smaller data sets that are typically more descriptive of a dimension in the fact table, and are frequently joined with fact tables. When performing joins, the local shard of a fact table on each node is joined with the local copy of the dimension table. We won’t create a dimension table in this example but we use them extensively in the client project I mentioned earlier in the post. In-terms of indexes, Firebolt has three types of index you can create to improve query performance: - **Primary Indexes**, mandatory with fact tables and optional for dimension tables, which physically sort and co-locate data to best align with how its queried, similar to clustering keys and clustered tables in Snowflake and BigQuery; the innovation here is in Firebolt’s use of sparse indexes that enable a much smaller and highly compressed range of data to be read from F3 into the engine cache at query runtime, producing query results faster due to the reduction in disk I/O - **Aggregating Indexes**, used for pre-calculating and storing the results of aggregate functions such as sum, count distinct and average. Think “materialized view” but using sparse indexes and other Firebolt technology designed to leverage the F3 storage format - **Join Indexes**, similar in purpose to the bitmap join indexes I used to use with Oracle data warehouses and designed to pre-calculate and store in-memory expensive table joins between fact and dimension tables. I’ll now create an aggregating index to go with my fact table, and then finally load data from the external table into my fact table ready for end-users to start querying my data. CREATE AND GENERATE AGGREGATING INDEX HACKER\_NEWS\_ALL\_AGG\_IDX ON HACKER\_NEWS\_ALL ( -- Add as many columns as required for the index -- Choose the columns that will be used in your group by statements month, title, author, type, sum(ranking), avg(ranking), count(distinct id), count(distinct author), avg(descendants) ); INSERT INTO HACKER\_NEWS\_ALL SELECT title , url , "text" , dead , "by" as author , score , "time" , "timestamp" , date\_trunc('month',"timestamp") as month , type , id , parent , descendants , ranking , deleted FROM HN\_FULL; Once all data has been loaded into the Firebolt fact table, setting up Looker to work with Firebolt is pretty straightforward and pretty-much the same process as when working with Snowflake, Redshift and BigQuery with a few exceptions at this point in time (for example, no support yet for symmetric aggregates) — see the [Looker docs](https://docs.looker.com/setup-and-management/database-config/firebolt) for the latest feature support details. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2Fd20d5-1cmo3dzd4njaaaxexpbvr_g.png) * * * Once my LookML views, model and explore are up and running its then just a case of creating a simple dashboard to see how fast it all runs. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F382b3-1ap9h2dkfpzk60lldnuru0q.png) And how fast did it all run? Queries generally took a second or two to respond with the dashboard feeling responsive and no significant delay in any tile returning data. As a comparison I created the same dashboard and LookML views, model and explore to run against the Google BigQuery dataset that I exported the data from originally, used each dashboard to run the same set of filter actions, dimension member selections and interactions and built a third dashboard that used the Looker System Activity explore to show response times of the two dashboards — I’ve made the Looker git repo containing these dashboards and Firebolt content available as a [public git repo on our Github account](https://github.com/rittmananalytics/firebolt_hn_looker). ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F03cf5-1_eghvewzf7az0svwgw44eg.png) So Firebolt was roughly 2x to 3x faster to return data than BigQuery in this very basic and rather unscientific comparison, and halving or more the response time of just this simple dashboard with a relatively small dataset and a single user isn’t something to be sniffed at. But where Firebolt really performs, as we’ve seen on the client project we’re currently implementing it for, is with datasets of terabyte and petabyte scale with lots of concurrent users and schemas containing multiple fact and dimension tables joined together as a dimensional model, or as a single fact table along with aggregating indexes designed to provide instant insights for embedded analytics within SaaS applications. For more insights on how we’ve used Firebolt keep an eye on this blog for further articles, and if you read this in time we’re also [speaking at the Budapest dbt Meetup](https://www.meetup.com/budapest-dbt-meetup/events/284955175/) in a couple of weeks time on using Firebolt with dbt — the event should be recorded and available later online and we’ll post details on how to watch after the event. ### Interested? Find out More Rittman Analytics is a boutique analytics consultancy specializing in the modern data stack who can help you get started with Firebolt and other cloud data warehouse platforms, [centralise your data sources](https://rittmananalytics.com/data-centralization) and [enable your end-users and data team](https://rittmananalytics.com/data-team-enablement) with best practices and a [modern analytics workflow](https://rittmananalytics.com/getting-started-with-dbt). If you’re looking for some help and assistance trialling and implementing Firebolt and building-out your analytics capabilities on a modern, flexible and modular data stack, [contact us now](https://calendly.com/markrittman/30min/?/?) to organize a 100%-free, no-obligation call — we’d love to hear from you! --- # Analyzing Drill to Detail Podcast Stats using Hex Canonical: https://rittmananalytics.com/blog/2022/04/22/analyzing-drill-to-detail-podcast-stats-using-hex URL: https://rittmananalytics.com/blog/2022/04/22/analyzing-drill-to-detail-podcast-stats-using-hex Description: How many listeners, subscribers and plays has the Drill to detail Podcast had since Episode 1 back in September 2016 with @stewartbryson? And as we near the 100th episode, what… Author: Mark Rittman Published: 2022-04-22T09:00:00+01:00 Modified: 2026-07-02T00:35:42+01:00 --- ![Analyzing Drill to Detail Podcast Stats using Hex](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F5f70e-1j7qh9e-d6iabdtdm60iyfg.jpeg) How many listeners, subscribers and plays has the [Drill to detail](https://twitter.com/drilltodetail) Podcast had since Episode 1 back in September 2016 with [@stewartbryson](https://twitter.com/stewartbryson)? And as we near the 100th episode, what were the Top 10 episodes of all-time? ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F5f70e-1j7qh9e-d6iabdtdm60iyfg.jpeg) I’ve crunched the numbers from Google Podcasts, iTunes Podcasts and Spotify Podcasts along with RSS feed subscriber data from our web hosts, Squarespace, and published the results in a [Hex](https://hex.tech/) notebook embedded below or accessible on the web [via this link](https://app.hex.tech/6163e97d-0d52-4d3b-829a-cd42b6be41ae/app/948afab5-bce9-40b7-92af-c7a8deb06db4/latest) — and congratulations to Kent Graziano, guest on our [most listened-to episode of all-time](https://rittmananalytics.com/drilltodetail/2019/7/22/drill-to-detail-ep71-the-rise-of-snowflake-data-warehouse-with-special-guest-kent-graziano)! --- # Rudderstack, Snowplow and Open-Source CDP Alternatives to Segment — Rittman Analytics Canonical: https://rittmananalytics.com/blog/2022/02/20/rudderstack-snowplow-and-open-source-cdp-alternatives-to-segment-rittman-analytics URL: https://rittmananalytics.com/blog/2022/02/20/rudderstack-snowplow-and-open-source-cdp-alternatives-to-segment-rittman-analytics Description: Rudderstack, Snowplow and Open-Source CDP Alternatives to Segment Marketing Analytics 20 Feb Written By Mark Rittman Segment is clearly the number one choice for organizations looking to add data collection,… Author: Mark Rittman Published: 2022-02-20T01:00:00+01:00 Modified: 2022-02-20T01:00:00+01:00 --- ![Rudderstack, Snowplow and Open-Source CDP Alternatives to Segment — Rittman Analytics](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2022%2F02%2F1752872305995-213ops.png%3Ffit%3D1400%252C1241%26ssl%3D1) # Rudderstack, Snowplow and Open-Source CDP Alternatives to Segment [Marketing Analytics](https://rittmananalytics.com/blog/category/Marketing+Analytics) 20 Feb Written By [Mark Rittman](https://rittmananalytics.com/blog?author=63c478ac6811232b3153d585) Segment is clearly the number one choice for organizations looking to add data collection, a customer data platform (CDP) and routing of that data to downstream marketing destinations, with the largest number of sources and destinations from any one vendor and options for schema governance, identity resolution, customer profiling and journey orchestration. Rittman Analytics is a Segment Select Partner and we’ve implemented Segment Connections, Segment Personas and Segment Protocols for a number of clients, setting up tracking plans and building customer data platforms that blend real-time behavioural data with data warehouse insights to build rich, contextual and persuasive customer experiences. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872305995-213ops.png%3Fresize%3D1400%252C1241%26ssl%3D1) And yet … Segment’s monthly user-based [pricing model](https://segment.com/pricing/) can become expensive if you’re running a fast-growing B2C website with rapidly increasing visitor counts who tend to browse once, rarely convert but quickly burn through your MTU quota (Monthly Tracked Users). And whilst Personas offers user-friendly audience building and journey orchestration tools, it’s API-based approach to data ingestion and limited user data model can be a [poor match to the way your organization actually stores and organises](https://www.fivetran.com/blog/customer-data-platform-data-warehouse) your customer data. So what about one of the open-source data collection and CDP alternatives to Segment such as Rudderstack or Snowplow? Are they as good, how do they work and why might you still want to stay using Segment, despite the cost? ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872306028-b9emcd.png%3Fresize%3D1400%252C742%26ssl%3D1) [Rudderstack](https://www.rudderstack.com/) is probably the simplest Segment alternative to explain; [an open-source re-implementation of Segment Connections](https://www.rudderstack.com/rudderstack-vs-segment/), the “classic” part of Segment that collects user event data from websites, mobile and other apps and then routes it to all of your downstream marketing and customer applications. Rudderstack’s Javascript SDK is API-compatible with Segment’s, [making migration just a case of](https://www.rudderstack.com/docs/user-guides/migration-guides/rudderstack-migration-guide/) finding all of the analytics.track and analytics.identify Javascript event calls on your website and appending rudder to each of them (or you can use the web snippet in our Github gist to [hijack the Segment SDK event calls and route them to Rudderstack](https://gist.github.com/markrittman/f9a4192dd0c803418c9cabfefcc603e3) as well). Anyone who’s familiar with Segment’s UI will be at home in Rudderstack’s, which is a bit less polished but improving fast and features a similar workflow to Segment’s for creating connections, viewing the live event stream and piping those events to your downstream destinations and cloud warehouse. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872306106-t1gu1e.png%3Fresize%3D1400%252C901%26ssl%3D1) What gets your attention initially with Rudderstack is its pricing model; what keeps that interest is it’s “warehouse-first” approach to storing, connecting and activating your first-party customer data. Segment’s Monthly Tracked Users (MTU) pricing approach means that a B2C eCommerce site with visitors that rarely convert, browse once and never return pays the same as a B2B site with high-value, logged-in frequently returning users that are far more likely to respond to personalized user journeys and precisely-targeted offers. Rudderstack’s hosted cloud option charges by the event instead of user, significantly reducing the penalty for sites with high bounce rates and low conversion rates. They do this is by not holding any of your event data in their own cloud storage, relying on you the customer streaming those events into your own cloud data warehouse. Which is great, in-fact, if like most businesses you’re looking to own your own first-party data and store it the same location as your legal jurisdiction. What’s not so great is [finding out that your backup has failed and then realising that Rudderstack doesn’t have a copy either](https://www.rudderstack.com/blog/protect-your-event-data-before-its-too-late/), so customers on Rudderstack’s top-end Enterprise Tier now have an [event replay facility](https://www.rudderstack.com/pricing/), essential for selling to organizations with deep pockets and a deep aversion to losing their customer data. You can only compete on price so long before another, even cheaper service comes along and then things become a race to the bottom. What makes Rudderstack interesting therefore to us as platform architects is their warehouse-centric approach to building your customer data platform, an architectural pattern we wrote about back last year in another blog post titled [“Why (and How) Customer Data Warehouses are the New Customer Data Platform”](https://rittmananalytics.com/blog/2021/3/12/customer-data-warehouses-are-the-new-customer-data-platform). ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872306137-llmjqn.png%3Fresize%3D1400%252C1266%26ssl%3D1) That blog was written in the context of [Hightouch](https://hightouch.io/), a [“reverse ETL”](https://rittmananalytics.com/blog/2021/2/15/customer-360-degree-analysis-and-hightouch) tool that enables organizations to take the rich, joined-up centralized records of their customers, their interests and insights from their behavioral patterns and sync them to their CRM, marketing and ad platform services. Rudderstack comes with its own reverse ETL tool, [Warehouse Actions](https://www.rudderstack.com/docs/warehouse-actions/), that whilst not quite as sophisticated as Hightouch, does the job and integrates with the rest of the Rudderstack suite of products. Rittman Analytics are delivery partners for both Rudderstack and Hightouch and each has its own place depending on the needs of our clients, and we use Warehouse Actions ourselves to keep our Hubspot CRM system up-to-date with insights we generate from visits to our website and form submissions. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872306162-78wbvh.png%3Fresize%3D1400%252C820%26ssl%3D1) Of course if we’re talking about open-source alternatives to Segment, the original grand-daddy of them all is [Snowplow Analytics](https://snowplowanalytics.com/), which like Rudderstack is also an open-source project with commercial sponsors and actually [predates Segment by a few months](https://news.ycombinator.com/item?id=4912076). Snowplow was created as a solution to a different problem than the one Segment originally addressed; [how to break free from the restrictions](https://snowplowanalytics.com/blog/2020/05/20/how-and-why-snowplow-was-founded/) that contemporary web analytics tools placed on accessing visitor behavioral data at the detailed, event-level, as opposed to Segment’s mission to [replace all of the individual web trackers on your site](https://news.ycombinator.com/item?id=4912076) with a single one that acted as a router to all of your downstream destinations. But in-practice, most organizations we see starting out with Segment, Rudderstack and Snowplow use it for one purpose first and foremost: capturing website, mobile app and web app user activity at the event level and storing it in a cloud data warehouse to combine customer data and analyze it using tools such as dbt and Looker. What makes Snowplow special though is the [emphasis it places on well-structured schemas](https://docs.snowplowanalytics.com/docs/understanding-tracking-design/understanding-schemas-and-validation/) for recording the details of your user events, and the lifecycle of those schemas as your users and their behaviours evolve. While Segment has [Protocols](https://segment.com/product/protocols/) to enforce schema governance after the event at heart it’s a service that favours flexibility over formality, allowing you to collect any event schema that conforms to its [track](https://segment.com/docs/connections/spec/track/) and other API specs. Snowplow’s approach makes it more likely that you’ll end up with user data that you can actually make sense of and derive meaning from, but at the cost of more initial work up-front to define those schemas and typically, a need for an IT department to host Snowplow and all its collectors, enrichers and other components for you and then manage, scale and monitor those comp Or, you could do as we’ve done and consume Snowplow as a managed service through another one of our partners, [SnowcatCloud](https://www.snowcatcloud.com/), who take care of all the hosting, configuration, scaling and monitoring for you; and in an age where [third-party cookies are disappearing](https://blog.chromium.org/2020/01/building-more-private-web-path-towards.html) and services like Google Analytics uses machine learning to fill-in the gaps in user data they no longer connect, SnowcatCloud’s new Snowplow-based [Iceberg](https://www.snowcatcloud.com/identity-graph/) identity graph service offers the promise of still being able to build a complete, 360-degree view of all customer interactions across all of their devices ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872306065-xzwm02.png%3Fresize%3D1184%252C710%26ssl%3D1) So is that it for Segment, have their customers all moved to cheaper or more specialised open-source products such as Rudderstack, Snowplow or even [Google Analytics 4](https://rittmananalytics.com/blog/2021/7/25/event-based-analytics-and-bigquery-export-comes-to-google-analytics-4-how-does-it-worknbsp-and-whats-thenbspcatch)? The answer, [according to IDC, is no](https://learn.segment.com/number1cdp/), and with Segment’s acquisition by Twilio and the launch of [Twilio Engage](https://techcrunch.com/2021/10/20/twilio-delves-more-deeply-into-marketing-with-new-tool-built-on-3-2b-segment-acquisition/?guccounter=1&guce_referrer=aHR0cHM6Ly93d3cucnVkZGVyc3RhY2suY29tLw&guce_referrer_sig=AQAAAHNwUFMAXcrgXJGqs0xxKVq6zQe8DcNHPrT3BTTcLto1Xv-64lItDBcjggPaYloRIMOyKMGp_6WkzK26Cdym9XFLAcT375ZizSOomODzOzn2rH6meK_PyyXm4KWF7vRoPPOSOr6Hy4HwVmwjHe5UQKw6xbh-Ob6DsN6cAsHnTE48) last year their strategy increasingly looks like one where they [move increasingly into the growth automation market](https://www.rudderstack.com/blog/how-twilio-is-shaping-segment/), marketing automation platforms powered by data pipelines and APIs. In the end it comes down to what are your needs, and at what stage in your data and marketing maturity is your organization currently at? [This report by the Customer Data Institute](https://www.cdpinstitute.org/wp-content/uploads/2021/07/CDPI-2289-Industry-Update-July-2021.pdf), I think, sums up the market in which Segment, Rudderstack, Snowplow and all of the other customer data platform and customer data infrastructure vendors operate: > *“This report groups CDP vendors into four categories based on the functions provided by their systems. Each category includes functions provided by the previous categories … Categories are:* > > ***Data CDP****. These systems gather customer data from source systems, link data to customer identities, assemble unified customer profiles, and store the results in a database available to external systems … In practice, these systems also can extract audience segments and send them to external systems. Systems in this category often employ specialized technologies for data management and access. Some began as tag management or Web analytics systems and retain considerable legacy business in those areas.* > > ***Analytics CDP.*** *These systems provide the features of a data CDP plus analytical applications. The applications always include customer segmentation and sometimes extend to machine learning, predictive modeling, revenue attribution, and journey mapping. These systems often automate the distribution of data to other systems.* > > ***Campaign CDP****. These systems provide data assembly, analytics, and customer treatments. What distinguishes treatments from segmentation is that treatments can be different for different individuals within a segment. Treatments may be personalized messages, outbound marketing campaigns, real time interactions, or product or content recommendations. These systems often include features to orchestrate customer treatments across channels.* > > ***Delivery CDP****. These systems provide data assembly, analytics, customer treatments, and message delivery. Delivery may be through email, Web site, mobile apps, CRM, advertising, or several of these. Products in this category often started as delivery systems and added CDP functions to support advanced analytics, personalization, or multi-channel campaigns.* Clearly, under this categorization, Rudderstack and Snowplow are Data CDPs and if combined with consulting services from specialist analytics partners such as Rittman Analytics, can become Analytics CDPs that provide a platform for segmentation, audience building and insights on your customers’ behaviours and needs. This in-fact describes the needs of most of our clients and we’ve implemented Rudderstack for a number of them, using savings made from their new license deal to pay for their CDP implementation. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872306132-czzmnk.png%3Fresize%3D1400%252C766%26ssl%3D1) But for clients further up the customer data platform maturity curve, Segment’s move into the Delivery CDP space means it continues to meet their needs as those needs increase in sophistication, scale and complexity; we still have many clients extremely happy with their Segment platform and others that are adopting it for the first time, and it’s still hands-down the best engineered, most fully-featured and user-friendly customer data infrastructure and platform that we work with today. Every client has their own particular needs and Segment, Rudderstack and Snowplow through SnowcatCloud offer different services meeting different needs for customers in different stages in their data maturity. We’re proud to partner with all three vendors and can advise on which best suits your needs, then work with your data team to build out the customer data platform that’s just right for you. ## Interested? Find out More Rittman Analytics is a boutique analytics consultancy specializing in the modern data stack who can help you get started with [Segment, Rudderstack, Hightouch or Snowplow](https://rittmananalytics.com/cdpsummary), [centralise your data sources](https://rittmananalytics.com/data-centralization) and [enable your end-users and data team](https://rittmananalytics.com/data-team-enablement) with best practices and a [modern analytics workflow](https://rittmananalytics.com/getting-started-with-dbt). If you’re looking for some help and assistance building-out your analytics capabilities on a modern, flexible and modular data stack, [contact us now](https://calendly.com/markrittman/30min/?/?) to organize a 100%-free, no-obligation call — we’d love to hear from you! [Customer Data Platform](https://rittmananalytics.com/blog/tag/Customer+Data+Platform)[Reverse ETL](https://rittmananalytics.com/blog/tag/Reverse+ETL)[Rudderstack](https://rittmananalytics.com/blog/tag/Rudderstack)[Segment](https://rittmananalytics.com/blog/tag/Segment)[Snowplow](https://rittmananalytics.com/blog/tag/Snowplow) [Mark Rittman](https://rittmananalytics.com/blog?author=63c478ac6811232b3153d585) --- # Lightdash, Looker and dbt as the BI Tool Metrics Layer — Rittman Analytics Canonical: https://rittmananalytics.com/blog/2022/02/01/lightdash-looker-and-dbt-as-the-bi-tool-metrics-layer-rittman-analytics URL: https://rittmananalytics.com/blog/2022/02/01/lightdash-looker-and-dbt-as-the-bi-tool-metrics-layer-rittman-analytics Description: Lightdash, Looker and dbt as the BI Tool Metrics Layer Modern Data StackLooker 1 Feb Written By Mark Rittman One of the more interesting startups to come out of the… Author: Mark Rittman Published: 2022-02-01T01:00:00+01:00 Modified: 2022-02-01T01:00:00+01:00 --- ![Lightdash, Looker and dbt as the BI Tool Metrics Layer — Rittman Analytics](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2022%2F02%2F1752872322031-k3jla.png%3Ffit%3D1400%252C1012%26ssl%3D1) # Lightdash, Looker and dbt as the BI Tool Metrics Layer [Modern Data Stack](https://rittmananalytics.com/blog/category/Modern+Data+Stack)[Looker](https://rittmananalytics.com/blog/category/Looker) 1 Feb Written By [Mark Rittman](https://rittmananalytics.com/blog?author=63c478ac6811232b3153d585) One of the more interesting startups to come out of the modern data stack space in the last twelve months is the team behind [Lightdash](https://www.lightdash.com/), an open-source alternative to [Looker](https://looker.com/) that uses [dbt](https://www.getdbt.com/), rather than [LookML](https://docs.looker.com/data-modeling/learning-lookml/what-is-lookml), to define its semantic model and metrics layer. Part of a wider trend within the modern data stack of [decomposing what historically were monolithic BI tool suites into smaller, more focused components](https://twitter.com/josephmoon_ai/status/1460468073522106369), Lightdash focuses on the end-user experience and leaves the definition of the underlying metrics layer to dbt; initially, through their own [specification of meta fields](https://docs.lightdash.com/references/metrics/) added to standard dbt projects but in the future, through dbt’s own emerging [metrics layer specification](https://github.com/dbt-labs/dbt-core/issues/4071). As well as being long-standing Looker and dbt partners we always like to check-out new products in the modern data stack, and so recently implemented Lightdash over a weekend as an internal R&D project on-top of our Google BigQuery and dbt-based internal data warehouse. Fast-forwarding to the tl;dr I think it’s fair to say that whilst Lightdash is still clearly a fairly primitive BI tool missing a host of features that we would consider table stakes in any serious BI tool evaluation, it’s integration with dbt and the benefits this provides to the developer experience, as well as its alignment with the trend in the industry towards BI tools using separate metrics layers, makes it worth keeping an eye on in-future. Here it is presenting our six major warehouse subject areas (“explores” in Looker lingo) and then displaying the various warehouse tables that can be joined-to from our main business operations clients table. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872322031-k3jla.png%3Fresize%3D1400%252C1012%26ssl%3D1) So how exactly does it integrate with dbt, and what’s involved in turning dbt into a BI tool’s metrics layer? [Installing and configuring](https://docs.lightdash.com/get-started/setup-lightdash/install-lightdash) Lightdash requires you either to host it yourself — we chose the one-click Heroku deployment option with a custom subdomain name that points to [https://lightdash.rittmananalytics.com,](https://lightdash.rittmananalytics.com,/) there are also options to run in Docker or [manually download and configure it all yourself](https://docs.lightdash.com/get-started/setup-lightdash/install-lightdash#deploy-manually) — or you can [join the waitlist](https://lightdash.typeform.com/to/HFlicx4i?typeform-source=hamzah905194.typeform.com) for the cloud-hosted, paid option, similar to other open-core/paid-hosted tools such as Superset, Airbyte and so on. A prerequisite for working with Lightdash is an existing dbt project at version 1.0.0 or higher; ours was at 0.21.1, the terminal release pre-1.0.0 but upgrading wasn’t a big deal and had to do be done at some point anyway, The way Lightdash works with dbt is for developers to add additional metadata for [dimensions](https://docs.lightdash.com/references/dimensions) and [metrics](https://docs.lightdash.com/references/metrics) onto existing [schema.yml](https://docs.getdbt.com/reference/configs-and-properties) entries for your dbt models, like this for a model column that will become a Lightdash dimension: – name: campaign\_datemeta:dimension:label: “Campaign Date”type: timestamptime\_intervals: \[‘raw’,’month’,’year’\] and this for a column that will become two Lightdash metrics: – name: total\_reported\_costmeta:dimension:label: “Total Reported Cost”type: numberhidden: yesmetrics:total\_total\_reported\_cost:type: sumlabel: “Total Total Reported Cost”avg\_total\_reported\_cost:type: averagelabel: “Average Total Reported Cost” Note that Lightdash is adding support for dbt-native metric definitions with the first, alpha-stage implemention detailed in the docs [here](https://docs.lightdash.com/references/metrics#2-using-dbts-metrics-tag). The issue we had was that not every model and column we wanted to see in Lightdash had been defined in schema.yml files in our dbt project, not ideal practice but also the reality in most dbt projects I’ve seen over the years. And then, of course, I needed to add potentially dozens of dimension and metric definitions to the schema file before I could start building reports and dashboards in the Lightdash tool. Whilst there are [a number of examples of SQL](https://github.com/dbt-labs/dbt-core/issues/1082) being used to auto-generate schema.yml files along with Python packages such as our own [droughty](https://github.com/dbt-labs/dbt-core/issues/1082), what I needed was something that also added the dimension and metric definitions to those schema entries. By using the datatype of each column and some basic column naming rules it should be possible to auto-generate at least a basic, starter version of a metrics layer for our warehouse that I could then fine-tune and enhance as time went on. I’ve shared the first pass at doing this in [a standalone git repositor](https://github.com/rittmananalytics/generate_lightdash_metrics_layer_4BQ)y which contains a single dbt macro called using dbt run-operation, along with parameter values for the warehouse schema, model prefix and primary key column prefix values (if needed). dbt run-operation generate\_metrics\_schema –args ‘{table\_schema: analytics, model\_prefix: wh\_, pk\_suffix: \_pk}’ The macro then outputs a SQL statement that you can run either through the BigQuery Web UI or bq CLI command to in-turn create the schema.yml file, or you can wrap both steps up into a single command that pipes each stages’ output into the next like this: dbt run-operation generate\_metrics\_schema –args ‘{table\_schema: analytics, model\_prefix: wh\_, pk\_suffix: \_pk}’ | \\tail -n +3 | \\bq query -use\_legacy\_sql=false -format sparse | \\tail -n +3 \> schema.yml Only thing then left to do is add my [join definitions](https://docs.lightdash.com/references/joins) into the schema definitions using a syntax very similar to LookML, like this: – name: wh\_ad\_campaigns\_dimmeta:label: “Wh Ad Campaigns Dim”joins:- join: wh\_ad\_campaign\_performance\_factsql\_on: ${wh\_ad\_campaigns\_dim.ad\_campaign\_pk} = ${wh\_ad\_campaign\_performance\_fact.ad\_campaign\_pk}- join: wh\_web\_sessions\_factsql\_on: ${wh\_ad\_campaigns\_dim.ad\_campaign\_pk} = ${wh\_web\_sessions\_fact.ad\_campaign\_pk} and ${wh\_ad\_campaign\_performance\_fact.campaign\_date} = date(${wh\_web\_sessions\_fact.session\_start\_ts})- join: wh\_web\_events\_factsql\_on: ${wh\_web\_sessions\_fact.web\_sessions\_pk} = ${wh\_web\_events\_fact.web\_sessions\_pk} I could also add custom measure and dimension definitions alongside the auto-generated ones to, for example, provide basic currency conversion for our revenue metric: – name: billing\_revenue\_gbpmeta:dimension:label: “Billing Revenue GBP”sql: “case when lower(${invoice\_status}) in (‘open’,’paid’) then case when ${invoice\_currency} = ‘USD’ then ${invoice\_local\_total\_revenue\_amount} \* 0.73 when ${invoice\_currency} = ‘CAD’ then ${invoice\_local\_total\_revenue\_amount} \* 0.57 when ${invoice\_currency} = ‘EUR’ then ${invoice\_local\_total\_revenue\_amount} \* 0.85 else ${invoice\_local\_total\_revenue\_amount} end end”type: numberhidden: yesmetrics:total\_billing\_revenue\_gbp:type: sum Next, I go into Lightdash and register the URL for my dbt git repo along with connection details for my BigQuery data warehouse. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872322151-mghk5.png%3Fresize%3D1400%252C869%26ssl%3D1) And once that’s done, I can start creating queries and building dashboards using dbt as the metrics layer for my BI tool rather than having to develop it all again in another BI tool and thereafter keep it in-sync with my dbt model. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872321863-eid4lb.png%3Fresize%3D1400%252C1017%26ssl%3D1) To be fair and realistic, it’s early days for Lightdash and the range of charts, configuration options and features within their metrics layer are still fairly primitive; [there’s no equivalent to LookML’s symmetric aggregates](https://github.com/lightdash/lightdash/issues/961) capability yet, for example, and no access grants, Liquid templating and so on. But the allure of having just one metadata store for both your warehouse transformations and metrics definitions is certainly a compelling one and we’ll be keeping an eye on Lightdash as the product, and the startup behind it, evolves — and contact us now if you’re looking at Lightdash and need a hand in evaluating it and getting it up and running for your organization. ## Interested? Find out More Rittman Analytics is a boutique analytics consultancy specializing in the modern data stack who can [get you started with Looker](https://rittmananalytics.com/data-analytics-main) (and Lightdash!), [centralise your data sources](https://rittmananalytics.com/data-centralization) and [enable your end-users and data team](https://rittmananalytics.com/data-team-enablement) with best practices and a [modern analytics workflow](https://rittmananalytics.com/getting-started-with-dbt). If you’re looking for some help and assistance building-out your analytics capabilities on a modern, flexible and modular data stack, [contact us now](https://calendly.com/markrittman/30min/?) to organize a 100%-free, no-obligation call — we’d love to hear from you! [Business Intelligence](https://rittmananalytics.com/blog/tag/Business+Intelligence)[Data Visualization](https://rittmananalytics.com/blog/tag/Data+Visualization)[Looker](https://rittmananalytics.com/blog/tag/Looker) [Mark Rittman](https://rittmananalytics.com/blog?author=63c478ac6811232b3153d585) --- # Using Looker to Analyze and Visualise Customer Concentration Canonical: https://rittmananalytics.com/blog/2021/12/16/using-looker-to-analyze-and-visualise-customer-concentration URL: https://rittmananalytics.com/blog/2021/12/16/using-looker-to-analyze-and-visualise-customer-concentration Description: When you’re running a consulting business one of the risks you need to be aware of is customer concentration, when one or two clients grow to become a majority of… Author: Mark Rittman Published: 2021-12-16T00:04:05+00:00 Modified: 2026-07-02T00:30:43+01:00 --- ![Using Looker to Analyze and Visualise Customer Concentration](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F08879-1yncwsrbmvppqxle2jri4vw.png) When you’re running a consulting business one of the risks you need to be aware of is *customer concentration*, when one or two clients grow to become a majority of your billing revenue and go from being your greatest opportunity to the thing that now keeps you awake at night, and not in a good way. What can initially seem like a good thing can end-up becoming the biggest threat to the survival of your business as they increasingly take-up all of your management and sales time, other clients can end-up being neglected and when that customer moves on, you’ve got a big hole in your billing to suddenly replace and a sales pipeline you’ve now got to rebuild, urgently. Defined as the percentage of total revenue that your biggest single account contributes, consulting firms typically aim to keep our client concentration below 25%; other businesses in other industries have different guidelines with [10% being suggested by Forbes](https://www.forbes.com/2009/04/15/biggest-business-risks-entrepreneurs-management-risk.html?sh=4386b1b6718b) as a sensible limit. As a consultancy focusing on the modern data stack we use Looker to monitor and visualize our own customer concentration in the form of two data visualisations; one that shows our customer concentration curve and another that shows the trend of customer concentration for our business over time. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F08879-1yncwsrbmvppqxle2jri4vw.png) * * * To visualize customer concentration as a time-series chart we first create an explore query that aggregates revenue by month, then pivots the results by client as shown in the explore screenshot below. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2Fcfa90-12xe1nfqqk3gvpqixj2e8gq.png) To calculate client concentration for each month we then add a Looker table calculation to this query that: 1. Ranks the revenue amount in each pivot column against the list ([**pivot\_row**](https://help.looker.com/hc/en-us/articles/360023789993-Aggregating-Across-Rows-Row-Totals-in-Table-Calculations)) of revenue columns for the row 2. Finds the pivot column with the highest ranked revenue amount ([**pivot\_where**](https://docs.looker.com/exploring-data/creating-looker-expressions/pivot_where)) and returns that revenue amount 3. Divides that highest-ranked revenue amount by the sum of the list of revenue columns to give us the client concentration percentage. pivot\_where( rank( ${projects\_invoiced.invoice\_gbp\_revenue\_amount}, pivot\_row( ${projects\_invoiced.invoice\_gbp\_revenue\_amount} ) )=1,${projects\_invoiced.invoice\_gbp\_revenue\_amount} ) / sum(pivot\_row(${projects\_invoiced.invoice\_gbp\_revenue\_amount})) Looker adds this table calculation as a single column to the end of each row, like this: ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F7c7be-1at8okx_tajbqwmqlsn5oza.png) If you then hide the revenue column and visualize the customer concentration calculation in the form of a line chart, and add range [reference lines](https://help.looker.com/hc/en-us/articles/360023727054-Creating-Reference-Lines-for-Charts-through-the-Visualization-Editor) that highlight the points at which customer concentration becomes a risk, you can visualize this time series calculation as shown in the screenshot below. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2Fbf36e-1wquit54uybvwwmxsofyara.png) * * * For the second chart, creating a [customer concentration curve](http://www.aboutretailaudit.ru/eng/concentrationcurve_eng.php) visualisation involves a bit more work as you need to [“dimensionalize”](https://help.looker.com/hc/en-us/articles/360041919653-How-to-Dimensionalize-a-Measure-in-Looker) the percentage of overall revenue that each client represents and the percentage of all clients that the client represents, and then use those dimensions as the axis of another line chart on which we then plot the count of customers at each axis intersection. We do this by adding a new [derived table](https://docs.looker.com/data-modeling/learning-lookml/derived-tables)\-based view to our Looker project that aggregates revenue by client and then calculates the percentage of overall revenue each clients’ aggregated revenue represents. view: dynamic\_company\_attributes { derived\_table: { sql: with metrics as (SELECT companies\_dim.company\_pk AS company\_pk, SUM(projects\_invoiced.invoice\_revenue\_gbp) AS customer\_ltv FROM \`analytics.companies\_dim\` AS companies\_dim LEFT JOIN \`analytics.invoices\_fact\` AS projects\_invoiced ON projects\_delivered.timesheet\_project\_pk = projects\_invoiced.timesheet\_project\_pk WHERE (projects\_invoiced.invoice\_status ) IN ('Open', 'Paid') and {% condition projects\_invoiced.invoice\_date %} projects\_invoiced.invoice\_sent\_at\_ts {% endcondition %} GROUP BY 1), pct\_of\_revenue as ( SELECT \*, round((percent\_rank() over (order by customer\_ltv))\*100) as pct\_of\_revenue FROM metrics ) select \* from pct\_of\_revenue;; } dimension: company\_pk { primary\_key: yes hidden: yes type: string sql: ${TABLE}.company\_pk ;; } dimension: pct\_of\_revenue { type: tier tiers: \[10,20,30,40,50,60,70,80,90,100\] style: integer group\_label: " Company Metrics" sql: ${TABLE}.pct\_of\_revenue ;; } Note also the [templated filter](https://docs.looker.com/data-modeling/learning-lookml/templated-filters) that enables us to dynamically change the period over which the percentage is calculated by reference to whatever time range the`projects_invoiced.invoice_sent_at_ts` field is filtered on. I then join this derived table view back to my main LookML explore, like this: label: " Business Operations" view\_label: " Companies" join: dynamic\_company\_attributes { view\_label: " Companies" sql\_on: ${companies\_dim.company\_pk} = ${dynamic\_company\_attributes.company\_pk} ;; relationship: one\_to\_one type: left\_outer } and then create a new explore query that 1. Returns, as rows, a count of clients broken-down by my new `pct_of_revenue` tier dimension 2. Pivots the results of the query by invoice year, so that I can visualize my client concentrate curve by year and see how it has changed over time 3. Calculates, using another table calculation, the running total percentage of all customers that each row in the revenue percentage tier contains: running\_total( (${companies\_dim.count} / sum(${companies\_dim.count})) ) Putting this all together gives us an explore query, results set and line chart visualization as shown the screenshot below, showing each year as a chart series and allowing us to compare our customer concentration curve from year-to-year. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F3ff57-1ape5xcu-f1-rqgmsfsciva.png) Finally, these two visualisations are then added to a new dashboard to which we then add a time range filter on the invoice date, giving us the ability to calculate client concentration over any arbitrary period of time. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2Fef9f8-1lgfzrrna_fxt2zl85vwfag.png) ### Interested? Find out More Rittman Analytics is an official Looker Consulting Partner ([we’re hiring!](https://careers.rittmananalytics.com/?_gl=1*142czix*_ga*ODEzNDg5MTQyLjE2MzI4NjQ5MDk.*_ga_M1P5TSNCVM*MTYzNzUxOTc1MS4yNy4xLjE2Mzc1MTk4MzIuNTA.)) with a team of certified LookML developers who can [get you started with Looker](https://rittmananalytics.com/data-analytics-main), [centralise your data sources](https://rittmananalytics.com/data-centralization) and [enable your end-users and data team](https://rittmananalytics.com/data-team-enablement) with best practices and a [modern analytics workflow](https://rittmananalytics.com/getting-started-with-dbt). If the idea of analysing and visualising client concentration for your business using Looker sounds interesting to you, or you’re looking for some help and assistance building-out your new customer data analytics platform on a modern, flexible and modular data stack, [contact us now](https://calendly.com/markrittman/30min/?) to organize a 100%-free, no-obligation call — we’d love to hear from you! --- # Adding Forecasting to your Looker Reports and Dashboards Canonical: https://rittmananalytics.com/blog/2021/11/21/adding-forecasting-to-your-looker-reports-and-dashboards URL: https://rittmananalytics.com/blog/2021/11/21/adding-forecasting-to-your-looker-reports-and-dashboards Description: A new feature introduced in Looker recently is Forecasting, giving you the ability now to forecast forward one or more measures in your look or dashboard based on historical data from… Author: Mark Rittman Published: 2021-11-21T19:57:03+01:00 Modified: 2021-11-21T19:57:03+01:00 --- A new feature introduced in Looker recently is [Forecasting](https://docs.looker.com/exploring-data/visualizing-query-results/forecasting-in-visualizations), giving you the ability now to forecast forward one or more measures in your look or dashboard based on historical data from your application or data warehouse. Forecasting is a [Looker Labs feature](https://docs.looker.com/admin-options/settings/labs) that has to be enabled by your Looker administrator before you can use it, and like all Labs features it could change or even not make it through to a future Looker release. That said, let’s take a look at how it works by using it to forecast future revenue for our company sales dashboard, that as a starting point shows revenue for the current year from January through to October, the month that we have complete numbers for. To create my first forecast of revenue for the next three months I start by pressing the **Forecast** button in the visualization’s menu bar, then select the measure I wish to forecast and the number of months I wish to forecast that measure over. Forecasts are typically expressed as a value with upper and lower bounds together with a confidence level in the prediction being within those bounds; Looker uses the [AutoRegressive Integrated Moving Average (ARIMA) algorithm](https://docs.looker.com/exploring-data/visualizing-query-results/forecasting-in-visualizations#how_forecasted_results_are_created_and_displayed) with a [Prediction Interval](https://docs.looker.com/exploring-data/visualizing-query-results/forecasting-in-visualizations#prediction_interval) setting that sets the confidence level of the prediction, giving you the ability to [trade-off preciseness and confidence in those precise numbers](https://docs.looker.com/exploring-data/visualizing-query-results/forecasting-in-visualizations#autoregressive_integrated_moving_average_\(arima\)_algorithm) being predicted by the algorithm. For now I’ll leave this and other settings at their default value and press **Run** to view the results of the forecast. As you can see from the screenshot below, based on the ten months of revenue number for the year to-date, Looker is forecasting consistent numbers for what’s left of this year and the start of next. Taking the Confidence Interval up to its maximum 99% and then down to the minimum 80% increases and decreases the gap between upper and lower forecast bounds, but the actual revenue numbers forecasted stay the same. However, I know that year-end and start of the New Year are often the slowest months for us and I’d expect revenue to drop, rather than stay constant, over the next three months of November, December and January. I know I’ve got three years of revenue data in my data warehouse, so I extend the time window for the look to now bring-in a further two years of data, and enable the [Seasonality](https://docs.looker.com/exploring-data/visualizing-query-results/forecasting-in-visualizations#seasonality) setting so that the forecast algorithm accounts for known cycles or repetitive data trends when producing my forecast. Now my forecast reflects the historic trends and seasonal cycles of increased and decreased demand that we’ve seen since starting the business three years ago. Looker’s forecasting algorithm now predicts a drop in revenue over the next three months, in-line with our team’s expectations. To make it easier to focus on just this year’s numbers, I use the **Limit Displayed Rows** option in the visualization Settings menu to display just the ten actual months and three forecasted months in the look’s chart. Or I can switch the visualization type from line to bar chart, or one of the other [visualization types that supports the Forecasting feature](https://docs.looker.com/exploring-data/visualizing-query-results/forecasting-in-visualizations#supported_visualization_types), if this would be a better way of communicating the range of values that the forecasting model has predicted for us. Finally, as our Looker model contains target as well as actuals and forecast revenue numbers, I use the [merged results](https://docs.looker.com/exploring-data/exploring-data/merged-results) feature to create and then overlay target data onto the visualization so that I can compare our predicted numbers against the targets we’d set for the business. If you’re trying this with your own target data, make sure you choose the targets query as the primary source query for the merge as otherwise your target data will also stop in October, the limit of your actuals data, rather than extend over the additional three month forecast period. # Interested? Find out More Rittman Analytics is an official Looker Consulting Partner ([we’re hiring!](https://careers.rittmananalytics.com/?_gl=1*142czix*_ga*ODEzNDg5MTQyLjE2MzI4NjQ5MDk.*_ga_M1P5TSNCVM*MTYzNzUxOTc1MS4yNy4xLjE2Mzc1MTk4MzIuNTA.)) with a team of certified LookML developers who can [get you started with Looker](https://rittmananalytics.com/data-analytics-main), [centralise your data sources](https://rittmananalytics.com/data-centralization) and [enable your end-users and data team](https://rittmananalytics.com/data-team-enablement) with best practices and a [modern analytics workflow](https://rittmananalytics.com/getting-started-with-dbt). If the idea of adding revenue and other forecasted data into your Looker dashboards sounds interesting to you, or you’re looking for some help and assistance building-out your new customer data analytics platform on a modern, flexible and modular data stack, [contact us now](https://calendly.com/markrittman/30min/?) to organize a 100%-free, no-obligation call — we’d love to hear from you! --- # Event-Based Analytics (and BigQuery Export) comes to Google Analytics 4; How Does It Work … and… Canonical: https://rittmananalytics.com/blog/2021/07/25/event-based-analytics-and-bigquery-export-comes-to-google-analytics-4-how-does-it-work-and URL: https://rittmananalytics.com/blog/2021/07/25/event-based-analytics-and-bigquery-export-comes-to-google-analytics-4-how-does-it-work-and Description: Google Analytics 4, the latest update to Google’s free web analytics service does away with the old Universal Analytics measurement model of users, sessions and hits and instead replaces it… Author: Mark Rittman Published: 2021-07-25T19:14:26+01:00 Modified: 2026-07-02T00:30:37+01:00 --- ![Event-Based Analytics (and BigQuery Export) comes to Google Analytics 4; How Does It Work … and…](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2Fee255-1dr2qyuduwninagluygb3vq.png) Google Analytics 4, the latest update to Google’s free web analytics service does away with the old Universal Analytics measurement model of users, sessions and hits and instead replaces it with the same type of visitor event-based model used by Segment, Snowplow Analytics and Mixpanel. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2Fee255-1dr2qyuduwninagluygb3vq.png) So how does Google Analytics 4 work and why has it changed, how do you query its event data in Google BigQuery and perhaps most importantly … what’s the catch? * * * [I blogged about using Google Analytics (GA) for event-based visitor analytics](https://rittmananalytics.com/blog/2018/11/12/event-level-digital-analytics-using-google-analytics-fivetran-bigquery-andnbsplooker) back in 2018 but in-practice using the Universal Analytics release of GA available back then for this sort of analysis had significant limitations compared to using services such as Mixpanel, Heap Analytics and Segment. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F25a71-1-2b33wr8fmas7hxeyd7x9q.png) Apart from the usual issues when working with the free version of GA such as visitor data being aggregated to session level, sometimes sampled and only being available via report exports using GA’s API and third-party tools such as Supermetrics and Stitch, custom events in Universal Analytics were always limited by the assumption that they were describing user interactions with objects on a web page or mobile app screen. Custom events limited to just that narrow scope only need a limited set of object properties for which you need to record a value — its label and categorisation, the action you took such as clicking or pressing it and the value you’d like to assign to a user taking that action, for example the average value of a lead being submitted or a form being completed. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F84eb4-1grceyivyomuo9fd7a_kckq.png) More general customer and product event-collection services such as Segment and Snowplow Analytics are designed to track every type of event across all digital customer touchpoints, as well as offline activities such as in-store purchases that may have been influenced by marketing content a visitors may have seen online. Google Analytics 3 moved more towards this kind of flexible event and parameters-based model with [Unified App and Web Analytics](https://blog.google/products/marketingplatform/analytics/new-way-unify-app-and-website-measurement-google-analytics/) and Google Analytics 4 completed the transition to instead use an event-based measurement approach, but with changes added to increase privacy protection and incorporate techniques such as cookie-less tracking. * * * Upgrading a Universal Analytics property to GA4 involves setting up a new Google Analytics property for your site, so we did just that and created a new set of events to mirror our existing GA3 and Segment events using Google Analytics web interface and Google Tag Manager. Similar to how Heap Analytics automatically tracks events without needing to write Javascript code, GA4’s optional Enhanced Measurement feature can auto-capture a number of common user events such as page views, link clicks, scroll events and form submissions without you having to add server-side or Javascript tracking event code to your website. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F60bd1-1rolj8sdnlax3qbo9azeb3a.png) Again similar to Heap Analytics you can then create additional derived events directly within Google Analytics from these base event types, for example defining a conversion event as a page view for the final page in your checkout process. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F8e9b3-16r4xm3hj1jkrahwtbrsy0q.png) For requirements beyond what’s possible within Google Analytics itself you can feed offline and server-to-server events directly into GA4 via HTTP requests using an updated version of the Google Analyics Measurement Protocol, for example in the example below to send an event to record an offline meeting being booked. { "client\_id":"2", "user\_id":"604356a0cd37ed00041ce1ab", "timestamp\_micros":"1627226029669000", "non\_personalized\_ads":true, "user\_properties":{ "email":{ "value":"[mark@rittmananalytics.com](mailto:mark@rittmananalytics.com)" } }, "events":\[ { "name":"meeting\_booked", "params":{ "meeting\_title":"test meeting", "meeting\_timestamp":1627226029669000 } } \] } Or you could use Google Tag Manager to define any type of user event you wish as pairs of triggers and Google Analytics 4 Event tags, with the payload from those event tags then being sent on to Facebook Ads, Google Ads and other downstream marketing services, similar to Segment’s Connections service. * * * From my perspective as an analytics engineer who uses Google BigQuery just about every day but could never afford to cost of a Google Analytics 360 license, the fact that Google Analytics 4 now also comes with real-time export of event data to BigQuery for free, without having to pay the $150k/year that Google charge for GA360 is just simply, amazing. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F322b9-1xjt4s8wuy0ywoltqi6v5aq.png) Export is limited to just one BigQuery dataset and there’s no option to export into Snowflake or Redshift but even so, this is a game changer and brings event-based visitor analytics within the reach of just about any website owner. Once you’ve configured Google Analytics 4 to export events to your Google Cloud Platform project, you’ll see a dataset created with your GA4 property ID as the dataset name’s suffix, and two tables, one for your daily exports that’s ingestion date-partitioned and another for real-time event exports that gets dropped and recreated at the end of each day. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2Ff9a6d-1onk1-ci11ohi3jwvgeshxa.png) If you’re familiar with the nested, repeated columns found in Google Analytics 360’s export schema you’ll initially feel familar with how Google Analytics 4 stores page views, sessions and other types of events within a single table, aligning the storage of your event data with the ideal requirements of Google’s global storage system, Colossus. But GA4’s export schema isn’t quite the same as the one used by GA360; instead of each row storing an individual user’s session with all of the page view, ecommerce and other hits nested as repeating sets of columns within those session rows, GA4’s export schema stores each user event as a separate table row and the parameters for those events stored as sets of repeating, key/value column pairs. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F78cdd-1vqjavjiiu0vrkaijixskpw.png) Querying this table schema to return a list of pages viewed along with the page paths and referrers for those events requires you filter on “page\_view” as the event type and then obtain the page title, page path and referrer using BigQuery Standard SQL’s UNNEST command, like this: SELECT event\_timestamp, (SELECT value.string\_value FROM UNNEST(event\_params) WHERE event\_name = 'page\_view' AND key = 'page\_location') AS page\_path, ( SELECT value.string\_value FROM UNNEST(event\_params) WHERE event\_name = 'page\_view' AND key = 'page\_title') AS page\_title, ( SELECT value.string\_value FROM UNNEST(event\_params) WHERE event\_name = 'page\_view' AND key = 'page\_referrer') AS referrer, user\_pseudo\_id, user\_id FROM \`ra-development.analytics\_277223877.events\_\*\` -- modify to your project WHERE event\_name = 'page\_view' ORDER BY event\_timestamp and returns a set of rows, one per page view event, like this: ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F02c6b-17muqxirwtvo-scflkxp-pg.png) Rather than having to sessionize each visitor’s events yourself as you do with Segment, events in GA4 are already assigned a session ID that maps to a “session\_start” event type containing event parameters for the source of the visitor event, details of the browser technology and the geo location of the visitor’s device as shown in the next query example. SELECT user\_pseudo\_id, TIMESTAMP\_MICROS(event\_timestamp) AS session\_start\_ts, CAST(LEAD(TIMESTAMP\_MICROS(event\_timestamp),1) OVER (PARTITION BY CONCAT(user\_pseudo\_id) ORDER BY event\_timestamp) AS timestamp) AS session\_end\_ts, (SELECT value.int\_value FROM UNNEST(event\_params) WHERE key = 'ga\_session\_id') AS session\_id, (SELECT value.int\_value FROM UNNEST(event\_params) WHERE key = 'ga\_session\_number') AS session\_number, (SELECT value.string\_value FROM UNNEST(event\_params) WHERE key = 'page\_referrer') AS referrer, (SELECT value.string\_value FROM UNNEST(event\_params) WHERE key = 'page\_location') AS landing\_page\_path, (SELECT value.string\_value FROM UNNEST(event\_params) WHERE event\_name = 'page\_view' AND key = 'page\_title') AS landing\_page\_title, traffic\_source.name, traffic\_source.medium, traffic\_source.source, CASE WHEN device.category = "desktop" THEN "desktop" WHEN device.category = "tablet" AND app\_info.id IS NULL THEN "tablet-web" WHEN device.category = "mobile" AND app\_info.id IS NULL THEN "mobile-web" WHEN device.category = "tablet" AND app\_info.id IS NOT NULL THEN "tablet-app" WHEN device.category = "mobile" AND app\_info.id IS NOT NULL THEN "mobile-app" END AS device, device.mobile\_brand\_name, device.mobile\_model\_name, device.mobile\_marketing\_name, device.mobile\_os\_hardware\_model, device.operating\_system, device.operating\_system\_version, device.vendor\_id, device.advertising\_id, device.language, device.is\_limited\_ad\_tracking, device.time\_zone\_offset\_seconds, device.browser, device.browser\_version, device.web\_info.browser, device.web\_info.browser\_version, device.web\_info.hostname FROM \`ra-development.analytics\_277223877.events\_\*\` s -- modify to your project WHERE event\_name = 'session\_start' order by 1,2 The full range of BigQuery Standard SQL functions and syntax is available for use when querying the GA4 export schema, with the example below using CTEs (common table expressions) and the new PIVOT function in Standard SQL to count the various event type occurences in each visitor session. WITH events as (SELECT user\_pseudo\_id, (SELECT value.int\_value FROM UNNEST(event\_params) WHERE key = 'ga\_session\_id') AS session\_id, (SELECT value.int\_value FROM UNNEST(event\_params) WHERE key = 'ga\_session\_number') AS session\_number, replace(lower(event\_name),' ','\_') event\_name, count(\*) events FROM \`ra-development.analytics\_277223877.events\_\*\` GROUP BY 1, 2, 3, 4), session\_events\_pivoted as ( SELECT \* FROM (SELECT \* FROM events) PIVOT (SUM(events) for event\_name IN('session\_start', 'page\_view', 'podcast\_played', 'pricing\_viewed', 'view\_case\_studies', 'file\_download', 'contact\_us', 'contact\_us\_clicked','cta\_pressed','about\_us\_viewed')) ORDER BY 1,2), session\_event\_counts as ( select \* except (session\_number), case when page\_view = 1 then true else false end as is\_bounced\_session, case when session\_number \> 1 then true else false end as is\_returning\_user, case when contact\_us \> 0 or contact\_us\_clicked \> 0 or cta\_pressed \> 0 then true else false end as is\_converting\_session from session\_events\_pivoted) select \* from session\_event\_counts Like Segment’s ability to stitch together the authenticated and unauthenticated browsing sessions for an individual visitor, GA4’s equivalent user\_id and pseudo\_user\_id user properties can be stitched together to provide a user’s complete conversion journey from first visit to checkout and register. SELECT DISTINCT user\_pseudo\_id as user\_pseudo\_id, LAST\_VALUE(user\_id ignore nulls) OVER ( PARTITION BY user\_pseudo\_id ORDER BY event\_timestamp ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING ) AS user\_id, MIN(event\_timestamp) OVER ( PARTITION BY user\_pseudo\_id ) AS first\_seen\_at, MAX(event\_timestamp) OVER ( PARTITION BY user\_pseudo\_id ) AS last\_seen\_at FROM \`ra-development.analytics\_277223877.events\_\*\` -- modify to your project And now that we can stitch complete individual conversion journeys together it doesn’t take much more to create Data Studio dashboards like the one below together, powered by a multi-touch attribution model written using just a single BigQuery Standard SQL query. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F41d16-1ndqicsal1lgp3gipv0_ynq.png) Example BigQuery Standard SQL queries for these and other common use-cases can be found on our Github Gists site with individual links below: - [GA4 page views, page paths and referrers](https://gist.github.com/markrittman/f100a26640e067b8d2b827457c3e4f40) - [GA4 sessions](https://gist.github.com/markrittman/c392156f9945c53b1923c41c06edb54f) - [GA4 event name and parameter combinations](https://gist.github.com/markrittman/b7a580b57c241a3b7445c6d0c9406e19) - [GA4 session event counts and other metrics](https://gist.github.com/markrittman/4dc1818ed9bc534b0076153a2bf598aa) - [GA4 visitor page view journey](https://gist.github.com/markrittman/f490e6fca741a1aabc99e9229f449688) - [GA4 multi-touch attribution model](https://gist.github.com/markrittman/5fa843ab279743771d71eab12b2d854c) * * * So what’s the catch? Like any service with a significant change to how it works internally you can expect there to be limitations, issues and bugs you’ll need to work around while the service settles down; GA is no different here and the [industry consensus is to tread warily](https://www.searchenginejournal.com/google-analytics-4-backlash/411392/) and not switch off your Universal Analytics properties just yet. A more fundamental, and from Google’s perspective deliberate difference between GA4 and services such as Segment and Snowplow is the focus on user privacy and the [inevitable disappearance of third-party cookies](https://digiday.com/marketing/winners-losers-fallout-googles-plan-drop-cookies/) as a means of tracking user activity. What this means for GA4 is the [introduction of cookie-less tracking](https://infotrust.com/articles/intro-to-cookieless-anonymous-tracking-in-google-analytics/) and the use of [machine learning to fill-in gaps in user data](https://blog.google/products/marketingplatform/analytics/new_google_analytics/) but how exactly this works and how reliable is the data it produces are yet to really be understood. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F419c7-0bkt6wqmg6vve-gqq.png) [IP addresses anonymization is enabled by default](https://support.google.com/analytics/answer/2763052?hl=en) and cannot be disabled, making plugins and filters such as Snowplow Analytics’ [IP Lookup Enrichment plugin](https://docs.snowplowanalytics.com/docs/enriching-your-data/available-enrichments/ip-lookup-enrichment/) impossible with GA4, although the latter does in-fact automatically geo-code events for you without the need for a plug-in. And finally of course — when a service is free there’s not much you can do if the provider of that service wants to [close your account for political or other reasons](https://programetrix.com/programetrix-blog/cannabis-cbd/cbd-search-and-social-ad-approval/), something a service provider such as Google with lots of government contracts might be under more pressure to do than a lower-profile supplier such as Segment or Snowplow. Sometimes paying for a service and owning your own data makes more sense than getting things for free but having no redress if things go wrong, and personally I’d be wary of ditching a paid service such as Segment with rock-solid infrastructure and a clear set of vendor incentives to keep providing you with a great service, but bearing this all in-mind and certainly as an upgrade from Google Analytics 3, this is an amazing upgrade to what was already a very popular and capable service. ### Interested? Find out More If the idea of analyzing your website traffic interactions at individual visitor level, mapping your customer journey and doing it all for free using Google Analytics sounds interesting to you, or you’re looking for some help and assistance building-out your new customer data analytics platform on a modern, flexible and modular data stack, [contact us now](https://calendly.com/markrittman/30min/?/?) to organize a 100%-free, no-obligation call — we’d love to hear from you! --- # RFM Analysis and Customer Segmentation — Rittman Analytics Canonical: https://rittmananalytics.com/blog/2021/06/20/rfm-analysis-and-customer-segmentation-rittman-analytics URL: https://rittmananalytics.com/blog/2021/06/20/rfm-analysis-and-customer-segmentation-rittman-analytics Description: RFM Analysis and Customer Segmentation using Looker, dbt and Google BigQuery Looker 20 Jun Written By Mark Rittman One of the simplest but most actionable ways that you can analyse… Author: Mark Rittman Published: 2021-06-20T01:00:00+01:00 Modified: 2021-06-20T01:00:00+01:00 --- ![RFM Analysis and Customer Segmentation — Rittman Analytics](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2021%2F06%2F1752872376602-c0lf7g.png%3Ffit%3D2500%252C1985%26ssl%3D1) # RFM Analysis and Customer Segmentation using Looker, dbt and Google BigQuery [Looker](https://rittmananalytics.com/blog/category/Looker) 20 Jun Written By [Mark Rittman](https://rittmananalytics.com/blog?author=63c478ac6811232b3153d585) One of the simplest but most actionable ways that you can analyse your customer base is to build an [RFM (Recency, Frequency, Monetary Value)](https://en.wikipedia.org/wiki/RFM_\(market_research\)) model and make that analysis actionable using a Looker dashboard, such as the one below based on the RFM dashboard we use for our consulting business. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872376602-c0lf7g.png%3Fresize%3D2500%252C1985%26ssl%3D1) rfm\_dashboard.png Most commonly used by our clients in the retail, marketplace, professional services and eCommerce businesses, RFM models score your customer base by three factors: ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872376738-vqn8zk.png%3Fresize%3D1400%252C567%26ssl%3D1) 1\_OhV1O0SLjibJk6EvwkVg5g.png 1. **Recency**: How recently a customer has made a purchase, either in months, days or weeks depending on your market’s typical purchase cycle 2. **Frequency**: How often a customer makes a purchase, typically measured over the twelve months leading up to each customer’s last purchase 3. **Monetary Value**: How much money a customer spends on purchase, either in-total or on-average over the same twelve month period By taking the values for each customer, bucketing them to produce a score from 1 (lowest) to 5 (highest) and then concatenating all three scores together you get an easy way to divide-up your customers into segments (or “RFM cells”; high-spending new purchasers (514, 5 for recency, 1 for frequency and 4 for monetary value), almost-lost but previously loyal customers (153, 1 for recency, 5 for frequency and 3 for monetary value) and so on. One of the most popular ways to visualize your customer RFM segments is by using a grid such as the one below, with each segment labelled and sized proportionate to the volume of customers each contains. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872376593-hhhjeb.png%3Fresize%3D2500%252C2000%26ssl%3D1) rfm\_grid.png RFM analysis is particularly useful for sales and customer teams needing to focus their limited time and money on those customers for whom a change in behavior — from lapsed to active, or first-time to repeat shopper for example — would have the most impact on your bottom line. For example, by focusing retention efforts on customers who used to be frequent, loyal and high-value purchasers (RFM segment 355) the revenue upside is obviously much greater than if they managed to retain customers who were previously infrequent low-value purchasers (RFM segment 132). Similarly, there’s little point in incentivising customers who are already your most loyal, frequent and high-spending customers (RFM segment 555) when spending those incentives on customers who’ve just made their first purchase and have the potential to also become loyal and valuable repeat customers (RFM segment 514, for example) Building an RFM model is often one of the first projects our clients’ data teams ask us to help deliver as the concept is straightforward, the data required has usually just been centralized and the output is immediately actionable by customer and marketing teams keen to focus their time on customer activity that has the greatest potential to retain or increase revenue. We’ve created an RFM model using the project timesheets and invoicing data we centralise in our Google BigQuery data warehouse and used it to create the RFM analysis dashboard shown in the screenshot at the start of this blog; in the rest of this blog I’ll walk through how we build the model using dbt and LookML and used the visualization features in Looker to help analyze and make the data actionable. For the purposes of the RFM model we created for our consulting business, we defined our Recency, Frequency and Monetary Value measures as: - **Recency**: the number of months since the last invoice raised for a customer - **Frequency** : the number of invoices raised for the customer in the 12 months leading-up to their last invoice - **Monetary Value** : the total value of invoices raised for the customer over the 12 months leading-up to their last invoice We already had timesheets, invoice and client spend data available in the cloud data warehouse we build using [dbt](https://rittmananalytics.com/getting-started-with-dbt) and our [RA Warehouse dbt package](https://rittmananalytics.com/ra-warehouse-for-dbt)[the new release](https://rittmananalytics.com/blog/2021/1/8/new-121-release-of-ra-data-warehouse-for-dbt-fivetran-bigquery-segment-and-now-snowflake-dw) of which I covered in our blog last year. The invoices\_fact table in the warehouse contained invoice\_created\_at\_ts, invoice\_amount and other columns sourced from Harvest together with first\_invoice\_month\_ts and months\_since\_first\_invoice columns created using SQL window (analytic) functions that we used as part of retention and customer lifetime value calculations. To enable us to calculate each clients’ recency score we added a column that calculated the number of months each invoice’s creation date was from today, using date functions from the [dbt\_utils](https://github.com/fishtown-analytics/dbt-utils) dbt package so that our code stayed portable across BigQuery, Redshift, Snowflake and the other cloud data warehouse platforms our clients work with. {{ dbt\_utils.datediff(‘invoice\_created\_at\_ts’,’current\_timestamp’,’MONTH’) }} as invoice\_months\_before\_now, For the frequency and monetary value RFM scores we needed two more fact table columns that gave us the date of the very last invoice for that client, along with another that told us how many months before that last invoice this particular invoice was created: {{ dbt\_utils.date\_trunc(‘MONTH’,’MAX(date(invoice\_created\_at\_ts))OVER (partition by c.company\_pkRANGE BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING)’) }} last\_invoice\_month,{{ dbt\_utils.datediff(‘invoice\_sent\_at\_ts’,’MAX(DATE(invoice\_sent\_at\_ts))OVER (PARTITION BY c.company\_pkRANGE BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING)’,’MONTH’) }} as months\_before\_last\_invoice, Moving over to our LookML project in Looker, I could then define a [LookML dimension](https://docs.looker.com/data-modeling/learning-lookml/lookml-terms-and-concepts) using the months\_before\_last\_invoice column along with a LookML measure that calculated the minimum for that measure to give us the invoice recency for each client. dimension: invoice\_months\_before\_now {type: numbersql: ${TABLE}.invoice\_months\_before\_now ;;}measure: min\_invoice\_months\_before\_now {type: minsql: ${invoice\_months\_before\_now} ;;} To count the number of invoices in the 12 months leading-up to their last invoice I created a dimension that filtered on the months\_before\_last\_invoice column and then used it to filter the invoice count, and also the revenue measures: dimension: is\_invoice\_in\_clients\_last\_12m {type: yesnosql: ${months\_before\_last\_invoice} < 12 ;;}measure: total\_invoice\_count\_in\_clients\_last\_12m {type: count\_distinctsql: ${TABLE}.invoice\_pk ;;filters: \[is\_invoice\_in\_clients\_last\_12m: “Yes”\]}measure: total\_invoice\_gbp\_amount\_in\_clients\_last\_12m {value\_format\_name: gbptype: sumsql: ${TABLE}.invoice\_local\_total\_revenue\_amount ;;filters: \[is\_invoice\_in\_clients\_last\_12m: “Yes”\]} Each client then needs to be scored from 1 to 5 on each of their RFM factors, scoring clients whose last invoice was in the current month as 5, in the last two or three months as a 4 and so on until clients with the highest number of months since their last invoice are scored as a five. Looker has several methods by which you can [bucket a column’s values in LookML](https://help.looker.com/hc/en-us/articles/360023900353-Bucketing-in-Looker) but these all rely on you already knowing what bucket definitions are appropriate for your data; as we didn’t know the distribution of our data we installed the [Histogram Visualization](https://github.com/llooker/viz-histogram-marketplace/blob/master/README.md) plug-in from the [Looker Marketplace](https://rittman.eu.looker.com/marketplace/view/viz-histogram) that lets us discover these boundaries interactively. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872376601-maxbxc.png%3Fresize%3D2500%252C1894%26ssl%3D1) marketplace.png Using this visualization we entered a range of values and settings for the different bucket bin definitions, starting with the default 3 bins for the minimum months since last invoice measure used for the recency score. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872376754-gdk9uo.png%3Fresize%3D2500%252C1262%26ssl%3D1) hist\_1.png Trying different options for maximum number of bucket bins, intervals between bins and set definitions for each bucket bin’s boundary we eventually settled on bin definitions that gave us a reasonably even distribution that matched customer value groupings we used elsewhere in the business. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872376850-ok23l.png%3Fresize%3D2500%252C1262%26ssl%3D1) hist\_2.png Now we knew what bucket boundaries to use, those measures were then used to create a LookML [Native Derived Table](https://docs.looker.com/data-modeling/learning-lookml/creating-ndts). view: rfm\_model {derived\_table: {explore\_source: companies\_dim {column: total\_invoice\_count\_in\_clients\_last\_12m { field: projects\_invoiced.total\_invoice\_count\_in\_clients\_last\_12m }column: total\_invoice\_gbp\_amount\_in\_clients\_last\_12m { field: projects\_invoiced.total\_invoice\_gbp\_amount\_in\_clients\_last\_12m }column: invoice\_months\_recency { field: projects\_invoiced.min\_invoice\_months\_before\_now }column: company\_pk { field: company\_deal\_value\_attribute.company\_pk }filters: {field: projects\_invoiced.invoice\_gbp\_revenue\_amountvalue: “\>0”}filters: {field: company\_deal\_value\_attribute.company\_pkvalue: “\-EMPTY”}derived\_column: rfm\_frequency\_score {sql: case when total\_invoice\_count\_in\_clients\_last\_12m between 0 and 1 then 1when total\_invoice\_count\_in\_clients\_last\_12m between 1 and 2 then 2when total\_invoice\_count\_in\_clients\_last\_12m between 2 and 5 then 3when total\_invoice\_count\_in\_clients\_last\_12m between 5 and 10 then 4when total\_invoice\_count\_in\_clients\_last\_12m \> 10 then 5end;;}derived\_column: rfm\_recency\_score {sql: case when invoice\_months\_recency between 0 and 5 then 5when invoice\_months\_recency between 5 and 10 then 4when invoice\_months\_recency between 10 and 15 then 3when invoice\_months\_recency between 15 and 20 then 2when invoice\_months\_recency \> 20 then 1end;;}derived\_column: rfm\_monetary\_value\_score {sql: case when total\_invoice\_gbp\_amount\_in\_clients\_last\_12m between 0 and 5000 then 1when total\_invoice\_gbp\_amount\_in\_clients\_last\_12m between 5000 and 10000 then 2when total\_invoice\_gbp\_amount\_in\_clients\_last\_12m between 10000 and 25000 then 3when total\_invoice\_gbp\_amount\_in\_clients\_last\_12m between 25000 and 50000 then 4when total\_invoice\_gbp\_amount\_in\_clients\_last\_12m \> 50000 then 5end;;}}dimension: company\_pk {hidden: yesprimary\_key: yeslabel: ” Companies Company Pk”}dimension: rfm\_frequency\_score {group\_label: “RFM Model”}dimension: rfm\_recency\_score {group\_label: “RFM Model”}dimension: rfm\_monetary\_value\_score {group\_label: “RFM Model”} Note that we also added derived columns to the NDT for the label definition for each of these buckets and another that concatenated all three scores together to provide the [RFM “cell” segment name](https://clevertap.com/blog/rfm-analysis/) (541, 143, 315 etc) but I’ve left these out for brevity; similarly, we added another set of derived columns to the NDT that took combinations of recency, frequency and monetary value scores and assigned them appropriate segment names, for example for the 555 RFM cell that scores highest on all three factors. derived\_column: rfm\_segment {sql: case when (case when invoice\_months\_recency between 0 and 5 then 1when invoice\_months\_recency between 5 and 10 then 2when invoice\_months\_recency between 10 and 15 then 3when invoice\_months\_recency between 15 and 20 then 4when invoice\_months\_recency \> 20 then 5end) = 5and (case when total\_invoice\_count\_in\_clients\_last\_12m between 0 and 1 then 1when total\_invoice\_count\_in\_clients\_last\_12m between 1 and 2 then 2when total\_invoice\_count\_in\_clients\_last\_12m between 2 and 5 then 3when total\_invoice\_count\_in\_clients\_last\_12m between 5 and 10 then 4when total\_invoice\_count\_in\_clients\_last\_12m \> 10 then 5end) = 5and(case when total\_invoice\_gbp\_amount\_in\_clients\_last\_12m between 0 and 5000 then 1when total\_invoice\_gbp\_amount\_in\_clients\_last\_12m between 5000 and 10000 then 2when total\_invoice\_gbp\_amount\_in\_clients\_last\_12m between 10000 and 25000 then 3when total\_invoice\_gbp\_amount\_in\_clients\_last\_12m between 25000 and 50000 then 4when total\_invoice\_gbp\_amount\_in\_clients\_last\_12m \> 50000 then 5end) = 5 then ’01: Best Clients’…end;;;} The final step in configuring the LookML project ready for building the dashboard was to join the LookML view containing our native derived table into our main LookML model, like this: explore: companies\_dim {label: “Business Operations”view\_label: ” Companies”join: rfm\_model {view\_label: ” Companies”sql\_on: ${companies\_dim.company\_pk} = ${rfm\_model.company\_pk} ;;type: left\_outerrelationship: one\_to\_one}…} so that the customer RFM fields it now provides are available alongside all of the other fields in our Looker explore, as you can see in the screenshot below. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872376780-7tmav.png%3Fresize%3D510%252C756%26ssl%3D1) explore.png So now that all the RFM scores and segment names have been configured in our LookML project, creating the dashboard itself is just a case of selecting the relevant explore fields and charting the resulting data using regular table and bar chart visualization types along with the [Packed Bubble](https://rittman.eu.looker.com/marketplace/view/viz-packed_bubble) visualization plug-in from the Looker Marketplace. We also used the Treemap visualization that you can manually install into your Looker instance from [Looker’s Custom Visualizations v2](https://github.com/looker/custom_visualizations_v2/blob/master/src/examples/treemap/README.md) Github repo and configured it to use the RFM segment cell name along with the count of clients in those segments as the measure. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872376783-e91bm8.png%3Fresize%3D1400%252C977%26ssl%3D1) 1\_HInXs9zbhjGeTEF1REZDfg.png It’s not an exact match to the RFM grid visualisation back at the start of this blog as there’s no way to specify frequency and monetary value as the visualization’s Y axis and recency as the X axis, but it does a good enough job for us in showing the ordering and relative size of each RFM cell segment. ## Interested? Find out More If the idea of creating an RFM model to segment your customer base sounds interesting to you, or you’re looking for some help and assistance building-out your new customer data analytics platform on a modern, flexible and modular data stack, [contact us now](https://calendly.com/markrittman/30min) to organize a 100%-free, no-obligation call — we’d love to hear from you! [Analytics](https://rittmananalytics.com/blog/tag/Analytics)[Customer Analytics](https://rittmananalytics.com/blog/tag/Customer+Analytics)[dbt](https://rittmananalytics.com/blog/tag/dbt)[Looker](https://rittmananalytics.com/blog/tag/Looker) [Mark Rittman](https://rittmananalytics.com/blog?author=63c478ac6811232b3153d585) --- # Adding Looker Regression Tests to dbtCloud CI/CD Pipelines using Spectacles Canonical: https://rittmananalytics.com/blog/2021/05/30/adding-looker-regression-tests-to-dbtcloud-ci-cd-pipelines-using-spectacles URL: https://rittmananalytics.com/blog/2021/05/30/adding-looker-regression-tests-to-dbtcloud-ci-cd-pipelines-using-spectacles Description: We’ve been users of the Looker testing tool Spectacles since its days as an open-source project started by Josh Temple and Dylan Baker, using it to check that Looker model… Author: Mark Rittman Published: 2021-05-30T14:40:07+01:00 Modified: 2021-05-30T14:40:07+01:00 --- ![Adding Looker Regression Tests to dbtCloud CI/CD Pipelines using Spectacles](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2021%2F05%2F0.08366951401619838.png%3Ffit%3D1600%252C731%26ssl%3D1) # We’ve been users of the Looker testing tool [Spectacles](https://www.spectacles.dev/) since its days as an [open-source project](https://github.com/spectacles-ci/spectacles) started by Josh Temple and Dylan Baker, using it to check that Looker model updates and new releases we push into production have had their [SQL content validated](https://docs.spectacles.dev/app/tutorials/validators#the-sql-validator) as well as its LookML, dashboards and looks. Most Looker developers will be familiar with the Validate LookML button that appears when you’re editing the LookML definition of your project, but this only checks that you’ve used the right syntax and property definitions for your dimensions and measures.  ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2F0.30981496186480617.png%3Fssl%3D1) What it doesn’t check is whether the SQL you’ve used is actually valid or whether the database objects that SQL relies on are still there and named as expected. Before Spectacles we’d try to do this by scheduling reports that selected against all of the fields in our explores but these only reported that a view had failed, not which field within the view, and was an inelegant solution. Spectacles lets you define Suites of [regression tests](https://en.wikipedia.org/wiki/Regression_testing) that run either on-demand, to a schedule or if you’ve licensed the Advanced or Elite editions of Looker, when one of your developers [submits a pull request](https://docs.looker.com/data-modeling/getting-started/git-options#integrating_pull_requests_for_your_project) for the git repo that hosts your LookML project.  ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2F0.1343848481996417.png%3Fssl%3D1) As well as testing that the SQL used in your Looker model is valid and all of the tables and columns it references are still present and correct, test suites can also run the [Content Validator](https://www.google.com/search?q=looker+content+validator&rlz=1C5CHFA_enGB947GB947&oq=looker+content+valida&aqs=chrome.0.0j69i57j0.2233j0j4&sourceid=chrome&ie=UTF-8) to check that your dashboards and looks aren’t referring to obsolete model content as well as any [Data Tests](https://docs.looker.com/reference/model-params/test) you’ve added to validate the model’s business logic. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2F0.7585090022086042.png%3Fssl%3D1) We use dbt (“Data Build Tool”) to build the underlying data warehouses for our client Looker projects, and use [dbtCloud’s CI/CD](https://rittmananalytics.com/blog/2019/6/11/continuous-integration-feature-branches-and-automated-build-tests-using-dbtcloud) (Continuous Integration/Continuous Deployment) feature to automatically stand-up a temporary BigQuery Dataset (equivalent to a schema in Snowflake or Redshift) on submission of a developer’s pull request, into which the package is test deployed to check that it compiles, runs and passes its internal checks before they’re allowed to be pushed into production. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2F0.2734654937116111.png%3Fssl%3D1) The catch, of course, is that whilst those dbt models and transformations may have passed our CI/CD test pipeline checks and deployed successfully into the production BigQuery dataset, those changes may still break Looker content that relied on tables or columns that may no longer be present.  Running a Spectacles test suite overnight each day will help identify those breaking changes after the event; the latest release of Spectacles goes one step further and lets us extend the CI/CD test pipeline initiated by dbtCloud to include any dependent Looker content.  Now we can test both the upstream database changes and downstream BI content in the one test pipeline, doing so in a temporary database dataset ensuring that users of our Looker dashboards aren’t affected by any last-minute working through of LookML changes we find we have to make to ensure the new release goes smoothly. Spectacles integration with dbtCloud’s CI/CD pipeline feature works by leveraging the same Github Actions feature that dbt uses to call an external web service; Spectacles adds a further step to the test pipeline that calls your regression test suite so that both need to complete successfully for your pull request merge checks to pass. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2F0.8426258588132524.png%3Fssl%3D1) Testing all of your Looker content in the same temporary dataset that dbtCloud uses for its test pipeline is achieved by passing the name of that dataset to Looker as a user attribute value using Looker’s API, and then using that user attribute to dynamically set the SQL data source for each of your LookML views. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2F0.1340609434520854.png%3Fssl%3D1) Test Suites in Spectacles now have an addition dbt button in their configuration settings for integration with dbtCloud pull requests. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2F0.8028190398516976.png%3Fssl%3D1) These new configuration settings let you tell Spectacles the URL for the dbt git repo that code that dbtCloud uses when deploying your test pipeline and the name of the user attribute variable you’re going to use to pass across the temporary dataset name. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2F0.40098208629008425.png%3Fssl%3D1) Now your dbt deployment test pipelines have to check whether they’ve broken any downstream Looker explores, looks, dashboards or data test assertions before they’re considered ready for deployment. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2F0.6987668395767688.png%3Fssl%3D1) Extending your dbt CI/CD pipeline in this way benefits both your end-users because your releases won’t break the Looker content they’ve come to rely on, and the analytics engineers in your data team who won’t be called on in the future so much to fix breaking changes they’ve inadvertently introduced into the data pipeline. Rittman Analytics is a Spectacles implementation partner along with partnerships with Looker, Fishtown Analytics (maintainers of dbt) and other modern data stack technology vendors. We help businesses adopt a modern data stack, build their analytics capability and deliver analytics projects that increase revenue and improve their operational efficiency.  If you’re interested in Spectacles, your analytics development workflow or any aspect of moving to a modern, flexible and modular data stack, [contact us now](https://calendly.com/markrittman/initial-discovery-call-with-mark-rittman) to organise a 100% free, no-obligation 30 minute call — we’d love to hear from you. --- # Why (and How) Customer Data Warehouses are the New Customer Data Platform Canonical: https://rittmananalytics.com/blog/2021/03/12/why-and-how-customer-data-warehouses-are-the-new-customer-data-platform URL: https://rittmananalytics.com/blog/2021/03/12/why-and-how-customer-data-warehouses-are-the-new-customer-data-platform Description: A couple of weeks ago I posted an article on this blog on Customer 360-Degree Analysis using BigQuery, dbt and new “reverse ETL” tools such as Hightouch and Rudderstack, and… Author: Mark Rittman Published: 2021-03-12T21:21:53+01:00 Modified: 2021-03-12T21:21:53+01:00 --- A couple of weeks ago I posted an article on this blog on [Customer 360-Degree Analysis using BigQuery, dbt and new “reverse ETL” tools](https://rittmananalytics.com/blog/2021/2/15/customer-360-degree-analysis-and-hightouch) such as Hightouch and Rudderstack, and [this article from Tejas Manohar on the Fivetran blog](https://fivetran.com/blog/customer-data-platform-data-warehouse) along with the latest episode of the Drill to Detail Podcast are great introductions to the concept of customer data warehouses as the new customer data platform. In this article I’m going to talk a bit more about we’ve transformed our customer data warehouse into the core of our new customer data platform, adding segmentation, interest scoring and marketing audiences that we then sync to services such as Facebook Custom Audiences and Intercom. We do this by creating a set of additional derived customer and contact tables that take data from all parts of our business, both offline and online, and use it to create a set of tables we use to drive our marketing activity. These additional warehouse tables contain sequenced details of each client’s journey from prospect to opportunity to engaged client, derive sets of behavioral traits we use for marketing personalization and categorise each of the contacts associated with those clients into one of several value segments. ## A Data Model for Customer Data Warehouse Data Platforms The data model diagram below shows these customer data platform tables and how they relate to the customer and contact dimensions in the warehouse: These derived tables include (links are to code in our [RA Warehouse for dbt](https://github.com/rittmananalytics/ra_data_warehouse) Github project, note that the specific examples in that package would need to be adapted for your particular data and requirements) - [CUSTOMER\_EVENTS\_XA](https://github.com/rittmananalytics/ra_data_warehouse/blob/master/models/warehouse/w_crm/bigquery/wh_customer_events_xa.sql), a derived table that takes each invoice, timesheet, sales opportunity and contact interaction for a client and turns them into a sequence of events that helps us understand the stage in the customer journey they’re at as well as their profitability at any particular point in time, for example as shown in the Looker visualization below: - [CONTACT\_WEB\_INTERESTS\_XA](https://github.com/rittmananalytics/ra_data_warehouse/blob/master/models/warehouse/w_crm/bigquery/wh_contact_web_interests_xa.sql), where we connect the page view activity recorded for each client contact when they visit our website directly or via links in marketing emails we send, tag those pages with category labels and then use when creating marketing audiences or in summary-level charts such as the one below for all contacts in-aggregate - [CONTACT\_SEGMENTS\_XA](https://github.com/rittmananalytics/ra_data_warehouse/blob/master/models/warehouse/w_crm/bigquery/wh_contact_segments.sql), where we take the influencer score described in our previous post and use it together with scores for engagement and stage in the buying cycle to place each of our contacts in a value segment as shown in the next Looker visualization and with the aim of our marketing being to move those contacts into segments of higher value over time - [CONTACT\_AUDIENCES\_XA](https://github.com/rittmananalytics/ra_data_warehouse/blob/master/models/warehouse/w_crm/bigquery/wh_contacts_audiences.sql), the most important table for marketing purposes that takes each of these interest scores, segments and other traits and records them alongside each contact email ready for syncing downstream to our marketing services. ## Audience Building and Downstream Syncing of Customer Traits Now we’ve pulled-together all of this valuable customer and contact marketing data, the next step is to sync it to our various downstream marketing and advertising technology services. For this we’ve been using a new service called [Hightouch](https://www.hightouch.io/) that I introduced in my previous blog, a tool that allows us to define selections of customer and contact data (or “models”) and then sync those models to various marketing and customer service applications. Creating a data synchronization job is a three-stage process: 1. Defining a model, a SQL query against the customer data warehouse tables that can either compute traits itself, similar to SQL traits and computed traits in Segment Personas, or can just query a table directly if you’ve already computed those traits as part of your warehouse build. 2\. Connecting to the service you want to sync data to. Hightouch connects via those services APIs and in my case I’ve added Intercom, Facebook Custom Audiences and Hubspot, plus Segment and Rudderstack’s Javascript APIs. When you sync contact and other data to event collection services such as Segment and Rudderstack they become either user identity events or in the case of Segment, segment entrance and exit track events, or you can send transaction-type data to these services as track events, something we’ve been doing to convert Jira issue data to a series of Segment events in our system. 3\. Finally you define the sync itself, which can be triggered either manually, on a schedule or after completion of a dbtCloud job run. ## Interested? Find out More If the idea of creating a customer data platform based around your existing data warehouse and being able to sync that information synced to your downstream marketing and customer applications, [contact us now](https://calendly.com/markrittman/initial-discovery-call-with-mark-rittman) to organise a 100% free, no-obligation 30 minute call to discuss your data needs — we’d love to hear from you. --- # Customer Data Warehouses are the New Customer Data Platform — Rittman Analytics Canonical: https://rittmananalytics.com/blog/2021/03/12/customer-data-warehouses-are-the-new-customer-data-platform-rittman-analytics URL: https://rittmananalytics.com/blog/2021/03/12/customer-data-warehouses-are-the-new-customer-data-platform-rittman-analytics Description: Why (and How) Customer Data Warehouses are the New Customer Data Platform Modern Data Stack 12 Mar Written By Mark Rittman A couple of weeks ago I posted an article… Author: Mark Rittman Published: 2021-03-12T01:00:00+01:00 Modified: 2021-03-12T01:00:00+01:00 --- ![Customer Data Warehouses are the New Customer Data Platform — Rittman Analytics](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2021%2F03%2F1752872407384-p8223q.jpg%3Ffit%3D1920%252C1080%26ssl%3D1) # Why (and How) Customer Data Warehouses are the New Customer Data Platform [Modern Data Stack](https://rittmananalytics.com/blog/category/Modern+Data+Stack) 12 Mar Written By [Mark Rittman](https://rittmananalytics.com/blog?author=63c478ac6811232b3153d585) A couple of weeks ago I posted an article on this blog on [Customer 360-Degree Analysis using BigQuery, dbt and new “reverse ETL” tools](https://rittmananalytics.com/blog/2021/2/15/customer-360-degree-analysis-and-hightouch) such as Hightouch and Rudderstack, and [this article from Tejas Manohar on the Fivetran blog](https://fivetran.com/blog/customer-data-platform-data-warehouse) along with the latest episode of the Drill to Detail Podcast are great introductions to the concept of customer data warehouses as the new customer data platform. In this article I’m going to talk a bit more about we’ve transformed our customer data warehouse into the core of our new customer data platform, adding segmentation, interest scoring and marketing audiences that we then sync to services such as Facebook Custom Audiences and Intercom. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872407384-p8223q.jpg%3Fresize%3D1920%252C1080%26ssl%3D1) reverse\_etl.jpg ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872407384-p8223q.jpg%3Fresize%3D1920%252C1080%26ssl%3D1) reverse\_etl.jpg We do this by creating a set of additional derived customer and contact tables that take data from all parts of our business, both offline and online, and use it to create a set of tables we use to drive our marketing activity. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872407599-h3pc28.png%3Fresize%3D2482%252C1172%26ssl%3D1) Screenshot 2021-03-12 at 21.01.05.png ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872407599-h3pc28.png%3Fresize%3D2482%252C1172%26ssl%3D1) Screenshot 2021-03-12 at 21.01.05.png These additional warehouse tables contain sequenced details of each client’s journey from prospect to opportunity to engaged client, derive sets of behavioral traits we use for marketing personalization and categorise each of the contacts associated with those clients into one of several value segments. ## A Data Model for Customer Data Warehouse Data Platforms The data model diagram below shows these customer data platform tables and how they relate to the customer and contact dimensions in the warehouse: ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872407627-rfh0ti.png%3Fresize%3D1600%252C1212%26ssl%3D1) 1\_zcbekcCWa\_-fofr7G0JtaA.png ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872407627-rfh0ti.png%3Fresize%3D1600%252C1212%26ssl%3D1) 1\_zcbekcCWa\_-fofr7G0JtaA.png These derived tables include (links are to code in our [RA Warehouse for dbt](https://github.com/rittmananalytics/ra_data_warehouse) Github project, note that the specific examples in that package would need to be adapted for your particular data and requirements) - [CUSTOMER\_EVENTS\_XA](https://github.com/rittmananalytics/ra_data_warehouse/blob/master/models/warehouse/w_crm/bigquery/wh_customer_events_xa.sql), a derived table that takes each invoice, timesheet, sales opportunity and contact interaction for a client and turns them into a sequence of events that helps us understand the stage in the customer journey they’re at as well as their profitability at any particular point in time, for example as shown in the Looker visualization below: ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872407608-rs9ohm.png%3Fresize%3D1862%252C892%26ssl%3D1) Screenshot 2021-03-12 at 21.43.12.png ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872407608-rs9ohm.png%3Fresize%3D1862%252C892%26ssl%3D1) Screenshot 2021-03-12 at 21.43.12.png - [CONTACT\_WEB\_INTERESTS\_XA](https://github.com/rittmananalytics/ra_data_warehouse/blob/master/models/warehouse/w_crm/bigquery/wh_contact_web_interests_xa.sql), where we connect the page view activity recorded for each client contact when they visit our website directly or via links in marketing emails we send, tag those pages with category labels and then use when creating marketing audiences or in summary-level charts such as the one below for all contacts in-aggregate ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872407651-pjvwfr.png%3Fresize%3D2500%252C1082%26ssl%3D1) Screenshot 2021-03-12 at 22.08.14.png ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872407651-pjvwfr.png%3Fresize%3D2500%252C1082%26ssl%3D1) Screenshot 2021-03-12 at 22.08.14.png - [CONTACT\_SEGMENTS\_XA](https://github.com/rittmananalytics/ra_data_warehouse/blob/master/models/warehouse/w_crm/bigquery/wh_contact_segments.sql), where we take the influencer score described in our previous post and use it together with scores for engagement and stage in the buying cycle to place each of our contacts in a value segment as shown in the next Looker visualization and with the aim of our marketing being to move those contacts into segments of higher value over time ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872407837-e3gbq.png%3Fresize%3D2500%252C751%26ssl%3D1) Screenshot 2021-03-12 at 22.26.33.png ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872407837-e3gbq.png%3Fresize%3D2500%252C751%26ssl%3D1) Screenshot 2021-03-12 at 22.26.33.png - [CONTACT\_AUDIENCES\_XA](https://github.com/rittmananalytics/ra_data_warehouse/blob/master/models/warehouse/w_crm/bigquery/wh_contacts_audiences.sql), the most important table for marketing purposes that takes each of these interest scores, segments and other traits and records them alongside each contact email ready for syncing downstream to our marketing services. ## Audience Building and Downstream Syncing of Customer Traits Now we’ve pulled-together all of this valuable customer and contact marketing data, the next step is to sync it to our various downstream marketing and advertising technology services. For this we’ve been using a new service called [Hightouch](https://www.hightouch.io/) that I introduced in my previous blog, a tool that allows us to define selections of customer and contact data (or “models”) and then sync those models to various marketing and customer service applications. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872407858-p32pj.png%3Fresize%3D1679%252C1491%26ssl%3D1) hightouch.png ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872407858-p32pj.png%3Fresize%3D1679%252C1491%26ssl%3D1) hightouch.png Creating a data synchronization job is a three-stage process: 1. Defining a model, a SQL query against the customer data warehouse tables that can either compute traits itself, similar to SQL traits and computed traits in Segment Personas, or can just query a table directly if you’ve already computed those traits as part of your warehouse build. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872407408-l7k63.png%3Fresize%3D2500%252C1940%26ssl%3D1) models.png ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872407408-l7k63.png%3Fresize%3D2500%252C1940%26ssl%3D1) models.png 2\. Connecting to the service you want to sync data to. Hightouch connects via those services APIs and in my case I’ve added Intercom, Facebook Custom Audiences and Hubspot, plus Segment and Rudderstack’s Javascript APIs. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872407837-yz328h.png%3Fresize%3D2500%252C632%26ssl%3D1) destinations.png ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872407837-yz328h.png%3Fresize%3D2500%252C632%26ssl%3D1) destinations.png When you sync contact and other data to event collection services such as Segment and Rudderstack they become either user identity events or in the case of Segment, segment entrance and exit track events, or you can send transaction-type data to these services as track events, something we’ve been doing to convert Jira issue data to a series of Segment events in our system. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872407624-boj4we.png%3Fresize%3D1226%252C966%26ssl%3D1) identify.png ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872407624-boj4we.png%3Fresize%3D1226%252C966%26ssl%3D1) identify.png 3\. Finally you define the sync itself, which can be triggered either manually, on a schedule or after completion of a dbtCloud job run. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872407599-fouuj8.png%3Fresize%3D2500%252C702%26ssl%3D1) syncs.png ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872407599-fouuj8.png%3Fresize%3D2500%252C702%26ssl%3D1) syncs.png ## Interested? Find out More If the idea of creating a customer data platform based around your existing data warehouse and being able to sync that information synced to your downstream marketing and customer applications, [contact us now](https://calendly.com/markrittman/initial-discovery-call-with-mark-rittman) to organise a 100% free, no-obligation 30 minute call to discuss your data needs — we’d love to hear from you. [Customer Analytics](https://rittmananalytics.com/blog/tag/Customer+Analytics)[Data Warehousing](https://rittmananalytics.com/blog/tag/Data+Warehousing) [Mark Rittman](https://rittmananalytics.com/blog?author=63c478ac6811232b3153d585) --- # Customer Cohorting, Retention Curves and Predictive Lifetime Value — Rittman Analytics Canonical: https://rittmananalytics.com/blog/2021/02/22/customer-cohorting-retention-curves-and-predictive-lifetime-value-rittman-analytics URL: https://rittmananalytics.com/blog/2021/02/22/customer-cohorting-retention-curves-and-predictive-lifetime-value-rittman-analytics Description: Customer Cohorting, Retention Curves and Predictive Lifetime Value using Looker and Google BigQuery Looker 22 Feb Written By Mark Rittman One of the dashboards I find most useful for understanding… Author: Mark Rittman Published: 2021-02-22T01:00:00+01:00 Modified: 2021-02-22T01:00:00+01:00 --- ![Customer Cohorting, Retention Curves and Predictive Lifetime Value — Rittman Analytics](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2021%2F02%2F1752872433965-kpmdk7.png%3Ffit%3D2500%252C1809%26ssl%3D1) # Customer Cohorting, Retention Curves and Predictive Lifetime Value using Looker and Google BigQuery [Looker](https://rittmananalytics.com/blog/category/Looker) 22 Feb Written By [Mark Rittman](https://rittmananalytics.com/blog?author=63c478ac6811232b3153d585) One of the dashboards I find most useful for understanding the direction of our business is the Customer Cohort Performance dashboard I’ve created using Looker, shown with demo numbers in the screenshot below. Across a number of key areas for our business such as efficiency of delivery, project profitability and client retention it tells me whether the customers we manage to attract and retain now are in-fact more, or less, valuable than the ones we acquired back in the past. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872433965-kpmdk7.png%3Fresize%3D2500%252C1809%26ssl%3D1) screencapture-rittman-eu-looker-dashboards-next-109-2021-02-22-01\_15\_39.png Why is this information important? Because on the one hand it tells me whether our spend on marketing and sales has delivered the increase in client spend and retention it was meant to and whether the delivery team were able to service those clients’ projects in an efficient manner. More importantly, hours to deliver and total client spend are leading and lagging indicators of how satisfied our clients are with our services, so if clients we acquire now are spending less and taking longer to service than before then there’s probably something wrong in how we’re currently running the business. So why are dashboards and visualisations that group customers into cohorts useful, and how do they differ from your usual time-series and overall customer spend dashboards? A simple time-series visualisation in Looker that shows client revenue per quarter might look something like the one in the screenshot below. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872434059-w2hv2f.png%3Fresize%3D1400%252C462%26ssl%3D1) 1\_FdyA2vspKOApYqRNSY7kiQ.png It’s telling me that revenue across all clients is generally increasing upwards quarter-by-quarter, but what it’s not telling me is where that revenue is coming from. I could add client name as a pivot into this visualisation, but with over three years of trading history in our data warehouse this area chart gets pretty busy. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872433992-olf6a.png%3Fresize%3D1400%252C462%26ssl%3D1) 1\_kmFLC-XEPpgK5NwQY\_s3OQ.png Turning the area chart into a bar chart and stacking the series makes the visualisation a bit easier to interpret than the area chart. What the visualisation is telling me now is, for each calendar quarter we’ve been in business, what proportion of the quarter’s business came from which particular customer. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872434020-8kxr1.png%3Fresize%3D1400%252C450%26ssl%3D1) 1\_WGIQN2KcPHe2T3cHqKHYoA.png But what it’s not telling me though is how much business each quarter’s new intake of customers brought in, and to do that we need two additional pieces of information; when the customer’s account was created and how long after that creation date did each customer transaction take place. The customer’s account creation date is a field you’ll usually find alongside other customer record fields in a Looker explore and by default it’ll be an actual date, for example 01-March-2020. For cohorting purposes though you’ll want this creation date to also be available as Month and Quarter of account creation, as we’ve done in our Looker explore. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872434050-xbj4ve.png%3Fresize%3D1270%252C1318%26ssl%3D1) 1\_JtvJ70HRQkSW93d-fBMYsQ.png To enable Month and Quarter of account creation in your explore make sure the account creation dimension in your Looker explore is configured as a **dimension\_group**, the underlying table column has a timestamp datatype (or you CAST the column into a timestamp datatype) and you specify **month** and **quarter** timeframes for the field. dimension\_group: company\_created {timeframes: \[date,month,quarter\]type: timesql: ${TABLE}.company\_created\_date ;;} Now instead of using customer name as the pivot in our original area chart visualisation we can pivot on the account creation quarter, like this: ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872434509-t8dzg.png%3Fresize%3D1400%252C472%26ssl%3D1) 1\_P47bVy9l59jUiq5gsnU4Lw.png Straightaway the chart is easier to read as we’ve grouped customer accounts into these calendar quarter groupings (or “cohorts”), with each cohorts’ initial and then subsequent revenue then shown as areas in the chart that flow from the initial quarter when they were first acquired. From the chart above we can see that the cohorts of customers we acquired in Q3–2018 and in 2019–04 have gone on to become our biggest spending and loyal customers, so we probably want to look at how we marketed and acquired customers back then to see who we were targeting and whether it’s worth repeating those campaigns.. Like many of our clients our business model is a subscription one, where clients sign-up for an initial number of delivery sprints that get their data stack built and their first couple of data sources centralised, then renew on a sprint-by-sprint basis as we add more data sources and start enabling their new data team hires. As with any subscription business then we need to measure and understand how long we’re retaining our customers and how much they’re worth to us, and when we know both of those things then we can start to estimate the lifetime value of each of those cohorts. To do this though we need to add a couple more pieces of information to the invoice or transactions data we hold for each of those customers, as well as the unique count of customers we have for invoices each quarter; the calendar quarter of that customers’ first invoice and the number of calendar quarters each invoice was from that first invoice’s calendar quarter, something you can do by extending the derived table definition for your invoices or transactions LookML view and using **DATE\_DIFF**, **DATE\_TRUNC** and the **MIN() OVER** analytic functions to calculate these two values for each invoice or customer transaction. view: invoices\_fact {derived\_table: {sql: SELECT \*,DATE\_DIFF(date(invoice\_sent\_at\_ts),MIN(date(invoice\_sent\_at\_ts)) OVER (PARTITION BY c.company\_pkRANGE BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING),QUARTER) AS quarters\_since\_first\_invoice,DATE\_TRUNC(MIN(DATE(invoice\_sent\_at\_ts)) OVER (PARTITION BY c.company\_pkRANGE BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING),QUARTER) AS first\_invoice\_quarterFROM invoices\_fact ;;} Now if I create a line chart visualization for all of our clients in aggregate, use quarters since first invoice as the X axis, count of unique customers as the measure and the following table calculation: ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872434004-opnqhtt.png%3Fresize%3D1400%252C589%26ssl%3D1) 1\_iTyi-3Sz2ck-6AqoL-4lvw.png what I now have is a decay curve for clients overall, telling me that 50% of all clients churn after their first quarter, 75% churn after two but those that survive end-up being customers for life. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872434057-yiadw8.png%3Fresize%3D1082%252C944%26ssl%3D1) 1\_JZPrw-V-lpVfYOnlbJVo9A.png If I add in quarter of first invoice as the customer cohort to this visualisation I can see, however, that clients we’ve acquired more recently are more likely (75%) to be retained than ones acquired when we first started out, indicating that our services now are more valuable or “sticky” over the long-term than they used to be. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872434114-2nzqts.png%3Fresize%3D1400%252C688%26ssl%3D1) 1\_ALt115MMvWiYSgqzfr5sMQ.png If we assume the average active client asks us to deliver two analytics delivery sprints each month at £5k/sprint and more recent cohorts of customers stay with us for three calendar quarters rather than two, that equates to an increase in customer lifetime value of £30k (3 months x 2 sprints x £5k), something I’d want to be aware of. Finally though, as not all customers and cohorts result in projects of equal value, taking the quarters since first invoice quarter and pivoting it in a table (legacy) visualisation, using quarter of first invoice as the table rows and sum of revenue as the measure shows us neatly how each cohort’s revenue has fallen away, or in some cases increased, each quarter after cohort acquisition. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872433832-4wcchi.png%3Fresize%3D1400%252C386%26ssl%3D1) 1\_kqtcpf8wVYgZ3ZH0f2RMdQ.png If you’re a subscription or product business looking to better understand the churn rate and value of customers you’re acquiring now compared to before and thereby the amount you can spend on acquiring more, [contact us now](https://calendly.com/markrittman/initial-discovery-call-with-mark-rittman) to organise a 100% free, no-obligation 30 minute call to discuss your data needs — we’d love to hear from you. [Cohorting](https://rittmananalytics.com/blog/tag/Cohorting)[Customer Analytics](https://rittmananalytics.com/blog/tag/Customer+Analytics)[Looker](https://rittmananalytics.com/blog/tag/Looker) [Mark Rittman](https://rittmananalytics.com/blog?author=63c478ac6811232b3153d585) --- # Customer 360 Data Warehousing and Sync to Hubspot using BigQuery, dbt, Looker and Hightouch — Rittman Analytics Canonical: https://rittmananalytics.com/blog/2021/02/15/customer-360-data-warehousing-and-sync-to-hubspot-using-bigquery-dbt-looker-and-hightouch-rittman-analytics URL: https://rittmananalytics.com/blog/2021/02/15/customer-360-data-warehousing-and-sync-to-hubspot-using-bigquery-dbt-looker-and-hightouch-rittman-analytics Description: Customer 360 Data Warehousing and Sync to Hubspot using BigQuery, dbt, Looker and Hightouch Marketing AnalyticsModern Data Stack 15 Feb Written By Mark Rittman Rittman Analytics are a Segment Implementation… Author: Mark Rittman Published: 2021-02-15T01:00:00+01:00 Modified: 2021-02-15T01:00:00+01:00 --- ![Customer 360 Data Warehousing and Sync to Hubspot using BigQuery, dbt, Looker and Hightouch — Rittman Analytics](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2021%2F02%2F1752872455378-8ndc1.png%3Ffit%3D1400%252C656%26ssl%3D1) # Customer 360 Data Warehousing and Sync to Hubspot using BigQuery, dbt, Looker and Hightouch [Marketing Analytics](https://rittmananalytics.com/blog/category/Marketing+Analytics)[Modern Data Stack](https://rittmananalytics.com/blog/category/Modern+Data+Stack) 15 Feb Written By [Mark Rittman](https://rittmananalytics.com/blog?author=63c478ac6811232b3153d585) Rittman Analytics are a Segment Implementation Partner and we use Segment Personas together with Segment Connections to [connect all of our digital marketing touchpoints](https://rittmananalytics.com/blog/2019/11/6/e058gepqwiyyx4mop3eb06mn9ckjkp) and [create a single view of client and visitor interactions](https://rittmananalytics.com/blog/2019/12/15/segmentcdpemailtracking) across those digital channels. Customer Data Platform services such as Segment Personas are great for collecting these granular interaction histories for clients and prospects, data that can be combined with transaction and other information we hold for those clients when they identify themselves to us when they login to our site or register their interest in a new product or service we’re offering. We’ve used customer data from our Segment Personas customer data to provide our Intercom chatbot service with information on the project we’re currently delivering for a client, their preferences and interests, making those interactions [more relevant, more efficient and more likely to result in a successful client outcome](https://rittmananalytics.com/blog/2020/7/17/connecting-intercom-to-segment-personas-for-more-relevant-and-cost-effective-customer-services-agents). ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872455378-8ndc1.png%3Fresize%3D1400%252C656%26ssl%3D1) 1\_uRNydys81LcioCfyLOSHOQ.png ## Why Do We Need a Customer Data Warehouse? Whilst Personas and customer data platforms in-general do a great job of understanding and better serving the needs of our clients and potential clients when they interact with the real-time, digital aspects of our business, by definition they only really understand and serve those clients and prospects who visit our website and other marketing channels. The majority of our client and prospect interactions happen offline with the sales, delivery and finance parts of our business and the data these interactions generate is much richer, diverse and wide ranging in-terms of scope and the years we’ve been in-business. The best place to hold this sort of “customer 360-degree” historical interaction data is a cloud data warehouse and in this blog I’ll show how we’ve used Google BigQuery together with Looker, Segment and our RA Warehouse for dbt to do just this, connecting offline and online data together for both clients and contacts and syncing that data to our Hubspot CRM system using a new “reverse ETL” service, [Hightouch](https://www.hightouch.io/). ## Creating a 360-Degree View of all Customer and Client Interactions I’ve blogged in the past about how we’ve built our in-house data warehouse using [Google BigQuery, dbt and our open-source RA Warehouse for dbt package](https://rittmananalytics.com/blog/2020/5/28/introducing-the-ra-warehouse-dbt-framework-how-rittman-analytics-does-data-centralization), now extended to [provide compatibility with Snowflake Data Warehouse](https://rittmananalytics.com/blog/2021/1/8/new-121-release-of-ra-data-warehouse-for-dbt-fivetran-bigquery-segment-and-now-snowflake-dw) and covering finance, projects, marketing, customer and product analytics. Data generated by our operations is centralized into a single integrated view of the business using [dbt (“Data Build Tool”)](https://rittmananalytics.com/getting-started-with-dbt) and our [RA Warehouse for dbt](https://rittmananalytics.com/ra-warehouse-for-dbt) package to connect, transform and orchestrate the various steps in the integration process; the screenshot below shows the number of sources and steps in the process currently taken to build-up our customer view with light-green steps being data sources and light-blue ones as transformations. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872455257-am208w.png%3Fresize%3D1400%252C842%26ssl%3D1) 1\_1zLMDIcQrz2mhmNw3CTv\_w.png The final output of this process is a BigQuery table that has one row per customer and then details of their billing, invoicing and associated contacts along with those contacts involvement in any marketing or sales activity, with the data structured as nested repeating BigQuery columns. Data from this table is then also used to create an event timeline table that shows the complete end-to-end history of our interactions with that client and all of their contacts, as shown in the table screenshots below. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872455147-3b9xq.png%3Fresize%3D1016%252C986%26ssl%3D1) 1\_nky3pyMrd2O-z7SUPqJsMQ.png Another set of steps in our dbt package combines and integrates data for individual contacts, combining their activity across all of the companies in our database they may have been associated with. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872459793-whvwqr.png%3Fresize%3D1400%252C867%26ssl%3D1) 1\_4I\_TwAMbRhCAHk\_Bbbz43w.png We then take these customer and contact datasets and use them to create customer and contact-centred explores in Looker; in the latest release of Looker there’s a new app you can install from the marketplace that lets you to [create interactive diagrams of your Looker explores](https://docs.looker.com/data-modeling/extension-framework/lookml-diagram) to show how views and models are linked together. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872455331-wirj2o.png%3Fresize%3D830%252C478%26ssl%3D1) 1\_L-rrejHUGE5IuH9ETqTUrQ.png The diagram below was created using this new feature and shows how we’ve configured the customer explore and how the various views over our warehouse tables connect to the central customer view. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872459924-184smk.png%3Fresize%3D1400%252C732%26ssl%3D1) 1\_C1RkpsKlIEmf0rM3sBcHkQ.png Similarly our contact-centred Looker explore connects all of our contacts’ activities together in one place, as opposed to being split amongst the multiple companies that contact might have been associated with over their time with us. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872455356-13du8j.png%3Fresize%3D1400%252C548%26ssl%3D1) 1\_Yyb87IakfJzXQ\_BucBhFBg.png As well as structuring and connecting our customer and contact data for analysis in Looker we also derive various analytical insights from these datasets, for example calculating the gross and net revenue for a client across all service channels or determining how influential a particular contact might be when they’re involved in a sales opportunity, something we do by quantifying their engagement with our team, use of Looker (via Stitch’s new [Looker metadata](https://www.stitchdata.com/integrations/lookml/) and [usage data](https://github.com/singer-io/tap-looker) integrations) and how many open and closed won opportunities they’ve been associated with. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872459852-dc5n1.png%3Fresize%3D1400%252C252%26ssl%3D1) 1\_Ro7OeJj0vZEgfoT-ozVeAA.png Insights such as these are useful to see in a Looker dashboard but become even more useful when synced into tools such as Hubspot used by our customer and sales team. ## “Reverse ETL” using Hightouch Whilst Hubspot and services such as Salesforce have public APIs that allow customer and contact records to be updated with data from the warehouse, doing this typically involves custom integration work and keeping up-to-date with (in Hubspot’s case, frequent) API spec changes. Services such as Stitch and Fivetran make setting up “ETL” data extract pipelines from SaaS applications like these easy and straightforward; a new category of “reverse ETL” services such as Hightouch and Census do this the other way around and make it easy to feed customer and other insights from your warehouse into downstream SaaS applications such as Hubspot , Facebook Audiences and Salesforce. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872455404-rgw3u4.png%3Fresize%3D1249%252C565%26ssl%3D1) 1\_LYiSeL9lG2dq88DdZhPO7g.png Setting up a sync of our contact influencer properties into Hubspot starts with my defining a query against our BigQuery warehouse to return the set of contacts, their unique Hubspot ID and the property values I want to sync into Hubspot CRM. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872455356-b7drth.png%3Fresize%3D1400%252C904%26ssl%3D1) 1\_X2eKw5iDA6-eJAkfS4luBw.png Prior to this I’d set-up connections to BigQuery and our Google Cloud Platform data warehouse, with options also to connect to Snowflake, Teradata, mySQL and other popular cloud data warehouses. For destinations there’s the usual options for ad networks (Google Ads, Facebook Custom Audiences etc) and marketing technology SaaS apps (Marketo, Salesforce Marketing Cloud, Hubspot and so on). ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872455353-34iiax.png%3Fresize%3D1244%252C1656%26ssl%3D1) 1\_EvC4G7vJ9xJW36S6aKUxqA.png If you’ve tried to feed new customer properties data into Salesforce or Hubspot from services such as Segment Personas you’ll know it’s a pretty hit-and-miss affair with docs out of date and APIs always changing. In contrast, reverse ETL services such as Hightouch make this task refreshingly simple and straightforward giving you flexibility over what contact property you use for merging incoming contact records and drop-down selectors to choose which query output columns are used to populate your standard and (pre-created) custom contact properties. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872455300-32sdt.png%3Fresize%3D1400%252C1400%26ssl%3D1) 1\_onYJJF1z0CxwqBeRTk2IyA.png Then I define what triggers the sync that Hightouch runs between my warehouse customer data and Hubspot; in my case it makes sense to have the completion of our dbt job run trigger the sync so I first go and register the API key for our dbtCloud account and then configure the sync job to trigger on completion of a specific dbtCloud job run. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872455365-c1htjo.png%3Fresize%3D722%252C666%26ssl%3D1) 1\_3B1Vdzrf8Vwef2NdQZFD4g.png I then trigger a test run of the dbt job and see that Hightouch then replicated those influencer scores from the warehouse tables that dbt populated for us. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872459831-hn2me.png%3Fresize%3D1400%252C953%26ssl%3D1) 1\_W9sWwsgECeKpwjRo\_zNW3Q.png Finally when I check Hubspot I can see the two new contact properties added to the relevant contact records. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872455435-zohm7w.png%3Fresize%3D1356%252C906%26ssl%3D1) 1\_L7zVCN2HSaeombvzqAtkRQ.png Similarly I can set up syncs of company-level data such as the gross and net revenue numbers for client billing, not normally available to Hubspot as a property value for company records. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872455327-e3ljw8.png%3Fresize%3D1118%252C888%26ssl%3D1) 1\_5ibyfKEwSmeB6bogUr8lxA.png ## Interested? Find out More If the idea of creating a complete, 360-degree view of your customers’ and contacts’ interactions with your business both online and offline and having that information synced to your downstream marketing and customer applications, [contact us now](https://calendly.com/markrittman/initial-discovery-call-with-mark-rittman) to organise a 100% free, no-obligation 30 minute call to discuss your data needs — we’d love to hear from you. [Customer Analytics](https://rittmananalytics.com/blog/tag/Customer+Analytics)[Customer Data Platform](https://rittmananalytics.com/blog/tag/Customer+Data+Platform)[Data Warehousing](https://rittmananalytics.com/blog/tag/Data+Warehousing)[dbt](https://rittmananalytics.com/blog/tag/dbt)[Looker](https://rittmananalytics.com/blog/tag/Looker) [Mark Rittman](https://rittmananalytics.com/blog?author=63c478ac6811232b3153d585) --- # Deduplicating Company Records in a Multi-Source Data Centralization Project — Rittman Analytics Canonical: https://rittmananalytics.com/blog/2021/01/17/deduplicating-company-records-in-a-multi-source-data-centralization-project-rittman-analytics URL: https://rittmananalytics.com/blog/2021/01/17/deduplicating-company-records-in-a-multi-source-data-centralization-project-rittman-analytics Description: Deduplicating Company Records in a Multi-Source Data Centralization Project using dbt, Google BigQuery or Snowflake Modern Data Stack 17 Jan Written By Mark Rittman One of the most common tasks… Author: Mark Rittman Published: 2021-01-17T01:00:00+01:00 Modified: 2021-01-17T01:00:00+01:00 --- ![Deduplicating Company Records in a Multi-Source Data Centralization Project — Rittman Analytics](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2021%2F01%2F1752872505888-vztl9t.png%3Ffit%3D1400%252C781%26ssl%3D1) # Deduplicating Company Records in a Multi-Source Data Centralization Project using dbt, Google BigQuery or Snowflake [Modern Data Stack](https://rittmananalytics.com/blog/category/Modern+Data+Stack) 17 Jan Written By [Mark Rittman](https://rittmananalytics.com/blog?author=63c478ac6811232b3153d585) One of the most common tasks in a [data centralization project](https://rittmananalytics.com/data-centralization) is to create single, deduplicated records for each of the companies, contacts, products and other entities the business interacts with. Doing this allows you to connect sales activity from Salesforce and Hubspot to project delivery and invoicing data from Jira and Xero, for example: ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872505888-vztl9t.png%3Fresize%3D1400%252C781%26ssl%3D1) cross-company.png Doing this in-practice can however get pretty confusing and complicated quickly as a [recent thread on the dbt Slack forum discussed](https://getdbt.slack.com/archives/C0VLZPLAE/p1610712146023700). To summarise the complications this adds to a typical data centralization project: - Assuming you’re going to create a single record for each company, contact, product and so on, how do you store the source system IDs for those entities in such a way that you can link the incoming deals, orders, invoices and payments from those source systems to these new records? - What happens if not all of the sources of company data, for example, provide all of the fields that the main source provides? A CRM source such as HubSpot will typically provide dozens of data fields for each company whereas others may only provide a name and billing address - What if one source has the concept of a company or organisation in how it works but doesn’t provide a table of companies in its data extract; for example when you’re working with Jira as a data source and each project is named after one of your clients, but there’s only project and issue tables in the data export and no explicit “company” table - And what if two of the companies, contacts or products you need to deduplicate don’t naturally merge together on the name, code or other field you use for this purpose? All of this is solvable with enough thinking, effort and coding but doing so whilst keeping your project agile, delivering fast and not becoming a spaghetti mess of hacked-together code can be a challenge. Here’s how we do this on our own client [data centralization projects](https://rittmananalytics.com/data-centralization), using Google BigQuery or Snowflake as the data warehousing platform, [dbt (“Data Build Tool”)](https://rittmananalytics.com/getting-started-with-dbt) and our [open-source RA Warehouse for dbt](https://rittmananalytics.com/ra-warehouse-for-dbt) framework [available on Github](https://github.com/rittmananalytics/ra_data_warehouse) as a public repo. ## Step 1 : Standardise Entity Staging Data Sources Customers, contacts, projects and other shared dimensions are automatically created from all data sources, deduplicating by name and merge lookup files using a process that preserves source system keys whilst assigning a unique ID for each customer, contact etc, as shown in the diagram below. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872505917-d3u8nw.png%3Fresize%3D1400%252C462%26ssl%3D1) method.png Each dbt staging module (a folder within a dbt project) provides a unique ID, prefixed with the source name, and another field value (for example, user name) that can be used for deduplicating dimension members downstream. WITH source AS ({{ filter\_stitch\_relation(relation=var(‘stg\_hubspot\_crm\_stitch\_companies\_table’),unique\_column=’companyid’) }}),renamed AS (SELECTCONCAT(‘{{ var(‘stg\_hubspot\_crm\_id-prefix’) }}’,companyid) AS company\_id,REPLACE(REPLACE(REPLACE(properties.name.value, ‘Limited’, ”), ‘ltd’, ”),’, Inc.’,”) AS company\_name,properties.address.value AS company\_address,properties.address2.value AS company\_address2,properties.city.value AS company\_city,properties.state.value AS company\_state,properties.country.value AS company\_country,properties.zip.value AS company\_zip,properties.phone.value AS company\_phone,properties.website.value AS company\_website,properties.industry.value AS company\_industry,properties.linkedin\_company\_page.value AS company\_linkedin\_company\_page,properties.linkedinbio.value AS company\_linkedin\_bio,properties.twitterhandle.value AS company\_twitterhandle,properties.description.value AS company\_description,CAST (NULL AS STRING) AS company\_finance\_status,cast (null as string) as company\_currency\_code,properties.createdate.value AS company\_created\_date,properties.hs\_lastmodifieddate.value company\_last\_modified\_dateFROMsource)SELECT\*FROMrenamed As long as a data source can provide an ID and a field by which you can connect its records to the ones from the other source systems, you can just pass null values for the ones that are missing, like this: WITH source AS ({{ filter\_stitch\_relation(relation=var(‘stg\_jira\_projects\_stitch\_projects\_table’),unique\_column=’id’) }}),renamed as (select \* from (SELECTconcat(‘{{ var(‘stg\_jira\_projects\_id-prefix’) }}’,replace(name,’ ‘,’\_’)) AS company\_id,name AS company\_name,cast (null as string) as company\_address,cast (null as string) AS company\_address2,cast (null as string) AS company\_city,cast (null as string) AS company\_state,cast (null as string) AS company\_country,cast (null as string) AS company\_zip,cast (null as string) AS company\_phone,cast (null as string) AS company\_website,cast (null as string) AS company\_industry,cast (null as string) AS company\_linkedin\_company\_page,cast (null as string) AS company\_linkedin\_bio,cast (null as string) AS company\_twitterhandle,cast (null as string) AS company\_description,cast (null as string) as company\_finance\_status,cast (null as string) as company\_currency\_code,cast (null as timestamp) as company\_created\_date,cast (null as timestamp) as company\_last\_modified\_dateFROM source ){{ dbt\_utils.group\_by(n=19) }})select \* from renamed If you’re wondering what [{{ filter\_stitch\_relation … }}](https://github.com/rittmananalytics/ra_data_warehouse/blob/master/macros/filter_stitch_relation.sql) is referring to, it’s a [dbt macro](https://docs.getdbt.com/docs/building-a-dbt-project/jinja-macros/) we use to filter out the historic (append-only) version of rows you get when using Stitch to replicate SaaS data sources into Google BigQuery: {%- macro filter\_stitch\_relation(relation, unique\_column) -%}SELECT\*FROM(SELECT\*,MAX(\_sdc\_batched\_at) OVER (PARTITION BY {{ unique\_column }} ORDER BY \_sdc\_batched\_at RANGE BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING) AS max\_sdc\_batched\_atFROM{{ relation }})WHERE\_sdc\_batched\_at = max\_sdc\_batched\_at{%- endmacro -%} ## Step 2 : Union Staging Data Sources Together A variable we’ve created within the [dbt\_project.yml](https://github.com/rittmananalytics/ra_data_warehouse/blob/master/dbt_project.yml.bigquery_example) lists of all the staging data sources that provide company records, named in a standard way so that if we prepend “stg” and append “\_staging” to the start of the data source name it equates to the name of the module (folder) within the dbt project. crm\_warehouse\_company\_sources: \[‘hubspot\_crm’,’harvest\_projects’,’xero\_accounting’,’stripe\_payments’,’asana\_projects’,’jira\_projects’ The [“Integration”](https://github.com/rittmananalytics/ra_data_warehouse/blob/master/models/integration/int_companies_pre_merged.sql) module for company data sources then uses this list of sources provided by the variable to drive a Jinja “for” loop that unions these sources together into a single CTE (common table expression) with t\_companies\_pre\_merged as ({% for source in var(‘crm\_warehouse\_company\_sources’) %}{% set relation\_source = ‘stg\_’ + source + ‘\_companies’ %}select'{{source}}’ as source,\*from {{ ref(relation\_source) }}{% if not loop.last %}union all{% endif %}{% endfor %}) ## Step 3 : Group and Deduplicate on a Common Field This unioned-but-not-yet-deduplicated set of all company records from all sources is then grouped (deduplicated) on a field that contains the same value for the company across all the different sources, such as the company name. grouped as (SELECTcompany\_name,max(company\_phone) as company\_phone,max(company\_website) as company\_website,max(company\_industry) as company\_industry,max(company\_linkedin\_company\_page) as company\_linkedin\_company\_page,max(company\_linkedin\_bio) as company\_linkedin\_bio,max(company\_twitterhandle) as company\_twitterhandle,max(company\_description) as company\_description,max(company\_finance\_status) as company\_finance\_status,max(company\_currency\_code) as company\_currency\_code,min(company\_created\_date) as company\_created\_date,max(company\_last\_modified\_date) as company\_last\_modified\_datefrom t\_companies\_pre\_mergedgroup by 1 ## Step 4 : Store IDs and Multi-Value Fields in Arrays For fields where we’d need to record a number of values for each company, for example when recording the various source system IDs we’ll need to have available when linking invoices coming from Xero, deals from Hubspot and payments from Stripe, we make use of [repeating columns](https://cloud.google.com/bigquery/docs/nested-repeated) when working with Google BigQuery target data warehouses, like this: all\_company\_ids as (SELECT company\_name, array\_agg(distinct company\_id ignore nulls) as all\_company\_idsFROM t\_companies\_pre\_mergedgroup by 1), and repeating nested columns (aka structs) when storing multi-field arrays of related values, for example when storing the components of an address: all\_company\_addresses as (SELECT company\_name, array\_agg(struct(company\_address,company\_address2,company\_city,company\_state,company\_country,company\_zip)ignore nulls) as all\_company\_addresses If the target warehouse is Snowflake then we’d achieve the same result by using a VARIANT data type to store the repeating fields as JSON. all\_company\_ids as (SELECT company\_name,array\_agg(distinct company\_id) as all\_company\_idsFROM t\_companies\_pre\_mergedgroup by 1),all\_company\_addresses as (SELECT company\_name,array\_agg(parse\_json (concat(‘{“company\_address”:”‘,company\_address,'”, “company\_address2″:”‘,company\_address2,'”, “company\_city”:”‘,company\_city,'”, “company\_state”:”‘,company\_state,'”, “company\_country”:”‘,company\_country,'”, “company\_zip”:”‘,company\_zip,'”} ‘))) as all\_company\_addresses ## Step 5: Use a Lookup File for Manual Entity Merging For dimensions where merging of members by name is not sufficient (for example, company names that cannot be relied on to always be spelt the same across all sources) we add a [seed file](https://github.com/rittmananalytics/ra_data_warehouse/blob/master/data/companies_merge_list.csv) to provide manual lookups to map one member to another via their source system IDs, for example: company\_id,old\_company\_idhubspot-3423423,xero-123121hubspot-2412121,stripe-214122xero-123121,salesforce-12312412 and then [extend the logic of the previous grouping step](https://github.com/rittmananalytics/ra_data_warehouse/blob/master/models/integration/int_companies.sql) to make use of this merge file, for example when BigQuery is the target warehouse: from companies\_pre\_merged cleft outer join (select company\_name,ARRAY(SELECT DISTINCT xFROM UNNEST(all\_company\_ids) AS x) as all\_company\_idsfrom (select company\_name, array\_concat\_agg(all\_company\_ids) as all\_company\_idsfrom (select \* from (selectc2.company\_name as company\_name,c2.all\_company\_ids as all\_company\_idsfrom {{ ref(‘companies\_merge\_list’) }} mjoin companies\_pre\_merged c1 on m.old\_company\_id in UNNEST(c1.all\_company\_ids)join companies\_pre\_merged c2 on m.company\_id in UNNEST(c2.all\_company\_ids))union allselect \* from (selectc2.company\_name as company\_name,c1.all\_company\_ids as all\_company\_idsfrom {{ ref(‘companies\_merge\_list’) }} mjoin companies\_pre\_merged c1 on m.old\_company\_id in UNNEST(c1.all\_company\_ids)join companies\_pre\_merged c2 on m.company\_id in UNNEST(c2.all\_company\_ids)))group by 1)) mon c.company\_name = m.company\_namewhere c.company\_name not in (selectc2.company\_namefrom {{ ref(‘companies\_merge\_list’) }} mjoin companies\_pre\_merged c2 on m.old\_company\_id in UNNEST(c2.all\_company\_ids))) The process is the same for Snowflake data warehouse targets albeit with slightly different SQL to reflect the different approach to repeating nested columns that this platform uses left outer join (select company\_name, array\_agg(all\_company\_ids) as all\_company\_idsfrom (selectc2.company\_name as company\_name,c2.all\_company\_ids as all\_company\_idsfrom {{ ref(‘companies\_merge\_list’) }} mjoin (SELECT c1.company\_name, c1f.value::string as all\_company\_ids from {{ ref(‘int\_companies\_pre\_merged’) }} c1,table(flatten(c1.all\_company\_ids)) c1f) c1on m.old\_company\_id = c1.all\_company\_idsjoin (SELECT c2.company\_name, c2f.value::string as all\_company\_ids from {{ ref(‘int\_companies\_pre\_merged’) }} c2,table(flatten(c2.all\_company\_ids)) c2f) c2on m.company\_id = c2.all\_company\_idsunion allselectc2.company\_name as company\_name,c1.all\_company\_ids as all\_company\_idsfrom {{ ref(‘companies\_merge\_list’) }} mjoin (SELECT c1.company\_name, c1f.value::string as all\_company\_ids from {{ ref(‘int\_companies\_pre\_merged’) }} c1,table(flatten(c1.all\_company\_ids)) c1f) c1on m.old\_company\_id = c1.all\_company\_idsjoin (SELECT c2.company\_name, c2f.value::string as all\_company\_ids from {{ ref(‘int\_companies\_pre\_merged’) }} c2,table(flatten(c2.all\_company\_ids)) c2f) c2on m.company\_id = c2.all\_company\_ids)group by 1) mon c.company\_name = m.company\_namewhere c.company\_name not in (selectc2.company\_namefrom {{ ref(‘companies\_merge\_list’) }} mjoin (SELECT c2.company\_name, c2f.value::string as all\_company\_idsfrom {{ ref(‘int\_companies\_pre\_merged’) }} c2,table(flatten(c2.all\_company\_ids)) c2f) c2on m.old\_company\_id = c2.all\_company\_ids) The deduplicated and grouped list of companies together with their multi-valued source IDs fields and other field groups, suitably manually and additionally merged using our manual merge lookup file are then joined together ready for loading into the companies warehouse dimension table. SELECT i.all\_user\_ids,u.\*,e.all\_user\_emailsFROM (SELECT user\_name,MAX(contact\_is\_contractor) as contact\_is\_contractor,MAX(contact\_is\_staff) as contact\_is\_staff,MAX(contact\_weekly\_capacity) as contact\_weekly\_capacity ,MAX(user\_phone) as user\_phone,MAX(contact\_default\_hourly\_rate) as contact\_default\_hourly\_rate,MAX(contact\_cost\_rate) as contact\_cost\_rate,MAX(contact\_is\_active) as contact\_is\_active,MAX(user\_created\_ts) as user\_created\_ts,MAX(user\_last\_modified\_ts) as user\_last\_modified\_ts,FROM t\_users\_merge\_listGROUP BY 1) uJOIN user\_emails eON u.user\_name = COALESCE(e.user\_name,’Unknown’)JOIN user\_ids iON u.user\_name = i.user\_name ## Step 6: Generate Surrogate key for Dimension Tab;e Then coming along at the last minute but taking all the glory, the [warehouse dimension table model](https://github.com/rittmananalytics/ra_data_warehouse/blob/master/models/warehouse/w_crm/wh_companies_dim.sql) step at the end adds a unique identifier for the company record using the dbt\_utils.surrogate\_key macro. WITH companies\_dim as (SELECT{{ dbt\_utils.surrogate\_key(\[‘company\_name’\]) }} as company\_pk,\*FROM{{ ref(‘int\_companies’) }} c)select \* from companies\_dim ## Step 7: Un-nest the Source ID Arrays for Fact Joins Finally, when populating the fact tables that join to this dimension you’ll need a way to join to those repeating and variant data type columns that contain the array of source system IDs for each company, as shown in the screenshot below for Google BigQuery target warehouses. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872505988-8oz0fc.png%3Fresize%3D1400%252C795%26ssl%3D1) array.png To do this when working with BigQuery as the target dimension we use the UNNEST() Standard SQL function to flatten those nested values and then perform the required fact table source to dimension table join, like [this example](https://github.com/rittmananalytics/ra_data_warehouse/blob/master/models/warehouse/w_projects/wh_delivery_projects_dim.sql) for our delivery projects fact table: WITH delivery\_projects AS(SELECT \*FROM {{ ref(‘int\_delivery\_projects’) }}),companies\_dim as (SELECT {{ dbt\_utils.star(from=ref(‘wh\_companies\_dim’)) }}from {{ ref(‘wh\_companies\_dim’) }})SELECT…FROMdelivery\_projects pJOIN companies\_dim cON p.company\_id IN UNNEST(c.all\_company\_ids) Snowflake performs the flatten of the repeated source system ID field using its own syntax, so that the equivalent SQL looks like this: companies\_dim as (SELECT c.company\_pk, cf.value::string as company\_idfrom {{ ref(‘wh\_companies\_dim’) }} c,table(flatten(c.all\_company\_ids)) cf)SELECT…FROMdelivery\_projects pJOIN companies\_dim cON p.company\_id = c.company\_id ## Interested? Find out More You can read more about our work with dbt, Google BigQuery, Snowflake and other modern data stack technologies on our website and blog: - [Rittman Analytics](https://rittmananalytics.com/home-index) homepage - [Data Centralization](https://rittmananalytics.com/data-centralization) consulting packages - [Rittman Analytics Blog dbt Posts](https://rittmananalytics.com/blog/tag/dbt) - [Contact Us](https://rittmananalytics.com/home-index/#about-us) How we handle deduplication and merging of company and contact data is just one of the design patterns we’ve included in our RA Warehouse for dbt framework that we’ve made available as free to use, open-source code in a public repo on our Github site. We’re a [dbt Preferred Consulting Partner](https://www.getdbt.com/ecosystem/) and this is one of the many ways we try and contribute value back to the dbt ecosystem, and we welcome issue reports, pull requests and suggestions on other problems encountered on data centralization projects that it’d be cool for us to solve using this framework. Or are you interested but don’t have the time to do it yourself? No problem! [Click here](https://calendly.com/markrittman/initial-discovery-call-with-mark-rittman) to book a 100% free, no-obligation 30 minute call to discuss your data needs and how we could help get your data centralization initiative moving now. [Data Engineering](https://rittmananalytics.com/blog/tag/Data+Engineering)[Data Warehousing](https://rittmananalytics.com/blog/tag/Data+Warehousing)[dbt](https://rittmananalytics.com/blog/tag/dbt) [Mark Rittman](https://rittmananalytics.com/blog?author=63c478ac6811232b3153d585) --- # New 1.2.1 Release of RA Data Warehouse for dbt, Fivetran, BigQuery, Segment (and now Snowflake DW!) Canonical: https://rittmananalytics.com/blog/2021/01/09/new-1-2-1-release-of-ra-data-warehouse-for-dbt-fivetran-bigquery-segment-and-now-snowflake-dw URL: https://rittmananalytics.com/blog/2021/01/09/new-1-2-1-release-of-ra-data-warehouse-for-dbt-fivetran-bigquery-segment-and-now-snowflake-dw Description: Last year I blogged about our RA Data Warehouse for dbt, a set of data models, data transformations and data warehousing design patterns for dbt (“Data Build Tool”), Fivetran, Stitch,… Author: Mark Rittman Published: 2021-01-09T02:10:48+01:00 Modified: 2021-01-09T02:10:48+01:00 --- Last year I blogged about our [RA Data Warehouse for dbt](https://rittmananalytics.com/blog/2020/5/28/introducing-the-ra-warehouse-dbt-framework-how-rittman-analytics-does-data-centralization), a set of data models, data transformations and data warehousing design patterns for dbt (“Data Build Tool”), Fivetran, Stitch, Segment, Google BigQuery that we use to rapidly build-out the data warehouse layer for analytics solutions and data platforms we build for our clients. *RA Warehouse for dbt Architecture* By pre-building data source modules for popular SaaS applications such as Xero, HubSpot, Salesforce, Google Ads and Stripe , connecting them to a common set of routines for combining, deduplicating and integrating each of their datasets we: - Deliver projects faster, more efficiently and at lower cost for our clients - Deliver work of higher quality, as we’ve already ironed-out all the bugs and refined the code over multiple engagements - Deliver more value by spending our time delivering analytics insights, rather than spending our clients’ budgets re-inventing the wheel - Provide integrated, best-practice datasets for [Marketing Attribution](https://rittmananalytics.com/marketing-attribution), [Product Analytics](https://rittmananalytics.com/product-analytics) consulting projects we deliver for clients *Our Conformed, Dimensional Warehouse Data Model* ### What Does the RA Warehouse for dbt Framework Contain? The core [RA Warehouse for dbt framework](https://github.com/rittmananalytics/ra_data_warehouse) we’ve open-sourced includes: - Pre-built, standardised data source models for popular SaaS applications (Hubspot, Xero, Facebook Ads, Segment etc) - Support for Stitch, Fivetran and Segment data pipeline services - Google BigQuery and Snowflake data warehouse compatibility - Data transformation and integration logic that combines multiple sources, deduplicates and creates single contact and company records - Subject-area dimensional warehouses e.g. Finance, Marketing, Product, CRM - Utilities for data profiling, ETL run logging and analysis - Simple configuration via settings in a single configuration file (dbt\_project.yml) ### What Use-Cases and Engagement Types does it Enable? Since our first blog on the framework last year we’ve since deployed and extended the framework on client engagements to: - Build a platform and marketing analytics platform for a UK Fintech / Gaming startup - Create a web analytics and marketing attribution solution for a US mobile app company - Deliver finance and customer analytics for an auction business - Deploy NetSuite ERP Retail, eCommerce and Wholesale analytics for a UK retailer - Calculate finance, marketing and customer lifetime value analytics for a UK mobile app business - Implement an operational analytics solution for a UK/US marketing technology startup as well as using it as the core of our own [internal operational analytics platform](https://rittmananalytics.com/blog/2019/5/7/how-rittman-analytics), example Looker screenshots below. *Screenshots from Rittman Analytics’ Internal Analytics Platform powered by dbt, BigQuery and Looker* ## New in the v1.2.0 Release – Support for dbt 17.0 and Snowflake Data Warehouse The recently released v1.2.0 release of the RA Warehouse for dbt framework introduces a number of improvements, extensions and new features including: 1. Refactored and updated data source and transformation logic including updates to reflect changes in dbt 17.0.x 2. Easier configuration and enablement of data sources and support for Fivetran and Segment as managed data pipeline technology 3. Support for Snowflake Data Warehouse as the target data warehouse platform See the [What’s New in Version 1.2.0](https://github.com/rittmananalytics/ra_data_warehouse/blob/master/whats_new_in_v_1_2_0.md) page in the repo documentation for full details. ## Interested? Find out More You can read more about our work with dbt, Google BigQuery, Snowflake and other modern data stack technologies on our website and blog: - [Rittman Analytics](https://rittmananalytics.com/home-index) homepage - [Data Centralization](https://rittmananalytics.com/data-centralization) consulting packages - [Rittman Analytics Blog dbt Posts](https://rittmananalytics.com/blog/tag/dbt) - [Contact Us](https://rittmananalytics.com/home-index/#about-us) We’ve open-sourced this framework to share our learnings and experience with the dbt and analytics engineering community, and to invite others to review, contribute and fork this repository – [go ahead and clone our repo now!](https://github.com/rittmananalytics/ra_data_warehouse/blob/master/README.md) Interested but don’t have the time to do it yourself? No problem! [Click here](https://calendly.com/markrittman/initial-discovery-call-with-mark-rittman) to book a 100% free, no-obligation 30 minute call to discuss your data needs and how we could help get your data analytics project moving now – **we’d love to hear from you**. --- # A New Look for Looker : Cross-Filtering, Interactive Dashboards, Aggregate Persistence and a New Data Visualization Marketplace Canonical: https://rittmananalytics.com/blog/2020/09/21/a-new-look-for-looker-cross-filtering-interactive-dashboards-aggregate-persistence-and-a-new-data-visualization-marketplace URL: https://rittmananalytics.com/blog/2020/09/21/a-new-look-for-looker-cross-filtering-interactive-dashboards-aggregate-persistence-and-a-new-data-visualization-marketplace Description: One of our most popular services is helping Looker customers update and refresh their Looker dashboards, especially if they’ve been using Looker for a while now and the team who… Author: Mark Rittman Published: 2020-09-21T20:48:21+01:00 Modified: 2020-09-21T20:48:21+01:00 --- One of our most popular services is helping Looker customers update and refresh their Looker dashboards, especially if they’ve been using Looker for a while now and the team who put it all together has since moved on. Back when Looker first came out it’s fair to say that it was back-end features such as direct SQL querying of column-store cloud data warehouses together with LookML that got the product noticed, not its data visualization and dashboarding features which lagged well behind desktop BI tools such as Tableau and Qlikview. What many customers don’t realise though is that Looker’s front-end features have had a serious overhaul over the past year, adding a whole new dashboard user experience along with a range of new data visualization types available through a marketplace. Let’s start though with the new dashboard experience which officially is an “experimental feature” but has been available in preview for a while now, presumably it’ll get an official release at the Looker Join remote keynote later this year. To enable the new dashboard experience you’ll need an admin to first turn the feature on globally using the **Admin** \> **Labs** \> **Experimental Features** settings page and then enable individual dashboards using the **Upgrade to Dashboards (Beta)** dashboard menu item, as shown below. The **New Dashboard Experience** comes with a set of new filter types including radio buttons, tag lists, checkboxes and drop-down menus all of which can be displayed in-line, as pop-ups or hidden behind a “more” option, a set of features we’ve recently added to our main web campaign analysis dashboard as shown below. Another new feature [just added to the Labs feature list](https://docs.looker.com/dashboards/cross-filtering) is **Cross-Filtering**. Again this has to be enabled by an admin and then individually for each dashboard, and make sure you read the docs around restrictions in this initial preview release, but once enabled you can click on segments within charts on your dashboard to filter all of the other charts by the value you’ve clicked on, as shown in the video below. Now that you’re starting to query Looker interactively, slicing dicing and filtering what can be seriously big datasets hosted in Google BigQuery, Snowflake or AWS Athena you might also want to start thinking about the [aggregate persistence feature](https://docs.looker.com/data-modeling/learning-lookml/aggregate_awareness) that’s also been introduced recently – contact us if so as we’ve been using it since it’s beta release last year. As well as new dashboarding features there’s a [new data visualization plug-in](https://docs.looker.com/data-modeling/marketplace) **[Looker Marketplac](https://docs.looker.com/data-modeling/marketplace)**[e](https://docs.looker.com/data-modeling/marketplace) that again, needs enabling by an admin as it’s still a beta feature under active development. The marketplace option we’re interested in is the Plug-ins one, shown below. Once enabled the Marketplace gives you access to a range of new visualization types such as the **Sankey Visualization**, used in the example below to help us explore the journeys visitors to our website make after clicking on one of our marketing campaign links. A variation on this is the **Collapsable Tree Visualization** type that lets you interactively explore the nodes in a network, something we use to explore and understand the web pages visitors to our site have shown interest in. The **Sunburst Chart Visualization** is a visually appealing way to showcase hierarchical data structures, or the steps in a visitor event journey as we’ve used it for in the example below. And … dashboard gauges come to Looker with the **Radial Gauge Visualization**, featuring tons of options around targets, fill types and value presentation. If you’re looking to refresh your Looker dashboards, add some of these new features and boost adoption of Looker within your organization, [book in a call using this link](https://calendly.com/markrittman/initial-discovery-call-with-mark-rittman) and we’d be pleased to help. All of our team are Certified Looker Developers and we’ve been beta testing these and other features for some time now, we’d love to share our learnings and see if we can help upgrade your Looker dashboard experience too! --- # Ad Spend and Campaign RoI Analytics — Rittman Analytics Canonical: https://rittmananalytics.com/blog/2020/09/19/ad-spend-and-campaign-roi-analytics-rittman-analytics URL: https://rittmananalytics.com/blog/2020/09/19/ad-spend-and-campaign-roi-analytics-rittman-analytics Description: Ad Spend and Campaign RoI Analytics using Segment, Looker, dbt and Google BigQuery Marketing Analytics 20 Sept Written By Mark Rittman You’ve probably heard of the saying “Half the money… Author: Mark Rittman Published: 2020-09-19T01:00:00+01:00 Modified: 2020-09-19T01:00:00+01:00 --- ![Ad Spend and Campaign RoI Analytics — Rittman Analytics](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2020%2F09%2F1752872521462-ie1s8j.png%3Ffit%3D1016%252C438%26ssl%3D1) # Ad Spend and Campaign RoI Analytics using Segment, Looker, dbt and Google BigQuery [Marketing Analytics](https://rittmananalytics.com/blog/category/Marketing+Analytics) 20 Sept Written By [Mark Rittman](https://rittmananalytics.com/blog?author=63c478ac6811232b3153d585) You’ve probably heard of the saying [“Half the money I spend on advertising is wasted; the trouble is, I don’t know which half.”](https://www.b2bmarketing.net/en-gb/resources/blog/half-money-i-spend-advertising-wasted-trouble-i-dont-know-which-half), and digital marketing was supposed to fix this problem. If only it were that simple. You spend money on online ads and know precisely who clicked on the advert, how many then went on to buy something (or “convert” in eCommerce terminology) and how many then went on to buy again and again and become your best customers.  Except you don’t. Because your ad spend data comes from the ad network and they’re attributing just about anyone who’s even been in the same room as one of your ads to their network as a conversion; you’ve got initial conversion revenue data in [Stripe](https://stripe.com/gb) and predicted lifetime value numbers in [Baremetrics](https://baremetrics.com/) but the chance of getting someone’s time in Engineering to connect it all together is about zero. Which means once again, another month comes around and you’re spending $50k, $60k, $100k on Facebook Ads, Google Ads and Bing Ads, you’ve got a target to hit for new customer acquisition … and you just don’t know which channels bring in your most profitable customers. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872521462-ie1s8j.png%3Fresize%3D1016%252C438%26ssl%3D1) pic1.png Previously on this blog I’d written about the revenue part of marketing attribution in [Multi-Channel Marketing Attribution using Segment, Google BigQuery, dbt and Looker](https://rittmananalytics.com/blog/2020/2/8/multichannel-attribution-bigquery-dbt-looker-segment), but what about the cost side? How can we bring in ad network cost data and use it, together with attributed revenue data, to work out how effective our ad campaigns have actually been? Take as an example a mobile app business that sells subscriptions via Stripe, uses Baremetrics to predict lifetime value for those subscriptions and uses Segment to collect user activity data across their mobile and web applications.  ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872521470-608ups.png%3Fresize%3D1600%252C603%26ssl%3D1) pci2.png Together with a cloud data warehouse service such as Google BigQuery and a data visualization tool such as Looker, we can put together a solution that lets us not only analyze the clicks and spend data from our ad networks, but also lets us connect that spend to the revenue we’re recording from Stripe and attributing to those ad campaigns to create dashboards like the one below. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872521511-if00ds.png%3Fresize%3D1600%252C717%26ssl%3D1) pic3.png Putting this solution together involves a series of steps that acquire, transform, join and combine data from all of these sources. 1. We first need to extract details of our ads, ad campaigns and the performance (clicks, spend, impressions etc) using APIs each of the ad networks provides for campaign analysis 2. Then we need to create a combined dataset from that ad network data that enables us to analyze spend and performance across all networks and campaigns 3. Next we bring the ad network data, typically only available in aggregated form at the ad, campaign and day level, and join it to the session and page view-level visitor event data we’re recording from our website and mobile app, enabling us to compare ad network reported numbers with the actual numbers we’re seeing from Segment 4. Separately to this ad spend analysis activity, we bring in data from the revenue attribution model described in [Multi-Channel Marketing Attribution using Segment, Google BigQuery, dbt and Looker](https://rittmananalytics.com/blog/2020/2/8/multichannel-attribution-bigquery-dbt-looker-segment) that assigns the credit for conversions, immediate and predicted lifetime revenue from those conversions to the sessions (touchpoints) that the converter interacted with on the path to conversion 5. Finally we aggregate the session-level data from the revenue attribution model up to the ad campaign and day level, and then join that revenue data to the ad network performance and observed clicks data so that we can report on the revenue earned and conversions resulting from the spend we’ve made on our individual ads and campaigns. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872521583-6ukpy.png%3Fresize%3D1400%252C792%26ssl%3D1) pic4.png Shown in the form of a data flow or data transformations graph with data sources on the left and the final transformed datasets on the right, the above steps would translate into the diagram above.  For the first step our usual approach to sourcing data from Facebook Ads and Google Ads would be to use a service such as Stitch or Fivetran to create a fully-managed data extract pipeline into Google BigQuery, but as [Segment Connections](https://segment.com/product/connections/) is already being used to bring in website and mobile app event data we can make use of another Segment feature, [Object Cloud App Sources](https://segment.com/docs/connections/sources/about-cloud-sources/), to replicate the Facebook and Google Ads data into BigQuery instead.  Object Cloud App Sources (as opposed to Cloud Event Sources and regular Event Sources) don’t register ad activity as events in Segment’s event stream but they do the same job as Stitch and Fivetran’s API extract data pipelines, and they’re included in the [entry-level Team Plan](https://app.segment.com/signup?plan=team-monthly) if you’re a paying Segment customer. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872521599-02ytgp.png%3Fresize%3D1600%252C655%26ssl%3D1) pic5.png Before we can combine data from the two ad network Segment sources for step 2 in our plan, we first have to standardise table and column naming across the two sources. We can do this using SQL SELECT statements either in SQL views or using a tool such as [dbt (“Data Build Tool”)](https://rittmananalytics.com/getting-started-with-dbt) that we use on most client analytics projects. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872521549-lrx90a.png%3Fresize%3D1600%252C938%26ssl%3D1) pic6.png Google Ads’ API provides spend and click performance data at each of the ad, ad group and campaign levels whereas Facebook’s API provides it at just ad-level, so if you want to base your reports and dashboards at the campaign level you’ll need to roll-up the Facebook data to campaign-level before joining it to Google Ads’ campaign performance data. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872521603-3i24tk.png%3Fresize%3D1600%252C1723%26ssl%3D1) pic7.png To enable comparison of this ad click and cost data from Facebook and Google to the actual clicks and revenue we’re recording via Segment from our website page view data we’ll need a way of connecting these two sets of data together. Facebook Ads provides [UTM parameter](https://en.wikipedia.org/wiki/UTM_parameters) values for your ads in the API tables it provides, making it easy to look for those values in the session entrance page view data Segment records when people clicking on those adverts arrive at your site. Google Ads [prefers that you use](https://support.google.com/google-ads/thread/1449463?hl=en) their encrypted gclid [(“Google Click Identifier”)](https://www.ppchero.com/what-the-gclid/) tracking codes rather than utm parameters when tracking campaign performance as they enable more campaign metadata to be included in tracking details and then encrypt it, stopping what could potentially be sensitive campaign names or other personally-identifiable data being sent in clear text in the URL querystring. However only Google Analytics can decrypt the contents of a gclid tracking code rendering them fairly useless when need that metadata to be readable by another process, for displaying the name of the campaign, ad or creative version in a non-GA dashboard report. As we want our ad spend analytics to be cross-platform and usable outside of GA we instead match on ad\_id and store it in the utm\_content parameter when setting up tracking codes for new ads, rolling the numbers up to ad group and campaign level from there when needed for reporting. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872521652-m3dfz.png%3Fresize%3D1600%252C749%26ssl%3D1) pic8.png Put together, the transformations for these first three steps in the process look like the subset shown below of the full transformation graph. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872521556-vq5w5f.png%3Fresize%3D1600%252C492%26ssl%3D1) pic9.png Each step in this graph is a SQL transformation needs to be executed in the right order, waiting for any other transformations the next step depends on to complete before handing-off control to the next stage, and so we use dbt to create a DAG (directed acyclic graph) of transformations to ensure everything gets executed in the correct order, and that we can test everything properly before rolling it out to end-users. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872521567-o5muh9.png%3Fresize%3D1600%252C535%26ssl%3D1) pic10.png You can read more about our use of this tool in [Introducing the RA Warehouse dbt Framework : How Rittman Analytics Does Data Centralization using dbt, Google BigQuery, Stitch and Looker](https://rittmananalytics.com/blog/2020/5/28/introducing-the-ra-warehouse-dbt-framework-how-rittman-analytics-does-data-centralization) and the code we use for ad spend analysis and marketing attribution is contained within the [RA Data Warehouse for dbt](https://github.com/rittmananalytics/ra_data_warehouse) public git repo on our Github site. Once we’ve run all of these ad spend and performance transformations and combined it with actual click data from our website through Segment, we end-up with a table that looks like the one below. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872521620-sjcu3f.png%3Fresize%3D1600%252C402%26ssl%3D1) pic11.png Using this data we can put together some useful charts and dashboards in Looker to show how spend on our advertising campaigns has translated into clicks and traffic sent to our website. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872521554-k4cc9x.png%3Fresize%3D1600%252C822%26ssl%3D1) pic12.png To calculate ratios and metrics such as [Return on Investment](https://support.google.com/google-ads/answer/14090?hl=en-GB#:~:text=How%20much%20profit%20you%27ve,%29%20%2F%20Cost%20of%20goods%20sold.) (RoI), [Return on Advertising Spend](https://www.bigcommerce.co.uk/ecommerce-answers/what-is-roas-calculating-return-on-ad-spend/) (ROAS) and our [LTV/CAC](https://corporatefinanceinstitute.com/resources/knowledge/valuation/cac-ltv-ratio/#:~:text=LTV%20stands%20for%20%E2%80%9Clifetime%20value,the%20cost%20of%20acquiring%20them.&text=Generally%20speaking%2C%20a%20ratio%20greater,that%27s%20not%20necessarily%20the%20case.) (Customer Lifetime Value / Customer Acquisition Cost) ratio, we’ll need to add ad campaign revenue data from the [Multi-Channel Marketing Attribution](https://rittmananalytics.com/blog/2020/2/8/multichannel-attribution-bigquery-dbt-looker-segment) model I mentioned right back at the start of this article. Our revenue attribution approach assigns a percentage of each conversion to sessions (touchpoints) leading up to the conversion (up to a maximum of 90 days before the conversion date) using first click, last click, even click and time-decay attribution models. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872521603-g8lple.png%3Fresize%3D1600%252C783%26ssl%3D1) ad13.png Time decay attribution splits the credit for conversions again across all qualifying sessions (touchpoints) but more favourably weights this credit towards more recent touchpoints using a formula where credit halves every 7 days (the “half-life”) using the formula 2^(-x/7) where x is the number of days prior to the conversion event. Expressed in BigQuery Standard SQL this formula comes out as shown below: CASEWHEN TIMESTAMP\_DIFF( session\_start\_ts, conversion\_ts, DAY ) \> -90THEN POWER( 2, TIMESTAMP\_DIFF( session\_start\_ts, conversion\_ts,DAY ) / 7 )END)/SUM (CASE WHEN TIMESTAMP\_DIFF( session\_start\_ts, conversion\_ts, DAY ) \> -90THEN POWER( 2, TIMESTAMP\_DIFF( session\_start\_ts, conversion\_ts, DAY ) / 7 )END )OVER (PARTITION BY blended\_user\_id, user\_conversion\_cycle )AS time\_decay\_attrib\_pct and a typical time series attribution looks like the screenshot below: ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872521550-ubb9md.png%3Fresize%3D1278%252C306%26ssl%3D1) ad14.png The final step in our series of transformations is to join the ad campaign spend and clicks data to the ad revenue attribution data, aggregated from the one-row-per-session (touchpoint) level of detail its stored in the attribution model to the same day/ad/campaign level as the ad campaign spend data. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872521599-4xxc9n.png%3Fresize%3D1128%252C1208%26ssl%3D1) pic15.png Now we’ve got ad spend and revenue linked together in this way, we can add a final set of metrics to our ad spend analytics dashboard and properly understand the actual return on investment we’re getting on our ad campaigns, enabling us to allocate budget to those campaigns that bring in the most conversions and customers with the greatest expected lifetime value. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872521574-d6zsvo.png%3Fresize%3D1600%252C431%26ssl%3D1) pic16.png So if you’re also looking to understand which half of your ad spend budget is being wasted, which ad channels and campaign approaches really bring in long term revenue and which ones just fill the funnel but never convert, then contact us now at [info@rittmananalytics.com](mailto:info@rittmananalytics.com) or [book in a free discovery call](https://calendly.com/markrittman/initial-discovery-call-with-mark-rittman) with Mark Rittman, author if this article and lets see if we can do the same for you and your business. [dbt](https://rittmananalytics.com/blog/tag/dbt)[Digital Analytics](https://rittmananalytics.com/blog/tag/Digital+Analytics)[Looker](https://rittmananalytics.com/blog/tag/Looker)[Marketing Attribution](https://rittmananalytics.com/blog/tag/Marketing+Attribution) [Mark Rittman](https://rittmananalytics.com/blog?author=63c478ac6811232b3153d585) --- # News on our Marketing Automation Quickstart Package for Segment and Intercom Canonical: https://rittmananalytics.com/blog/2020/07/16/news-on-our-marketing-automation-quickstart-package-for-segment-and-intercom URL: https://rittmananalytics.com/blog/2020/07/16/news-on-our-marketing-automation-quickstart-package-for-segment-and-intercom Description: Two trends are emerging clearly from the events of 2020 and the COVID-19 global pandemic; businesses and their customers are moving online, and those businesses need to find, market to… Author: Mark Rittman Published: 2020-07-17T00:58:21+01:00 Modified: 2020-07-17T00:58:21+01:00 --- Two trends are emerging clearly from the events of 2020 and the COVID-19 global pandemic; businesses and their customers are moving online, and those businesses need to find, market to and retain their customers in the most cost-efficient and effective way possible. ## Automate your Customer Service Interactions Intercom is an in-app “chatbot” messaging service we and many other online businesses use to automate the process of qualifying sales leads, answering support questions and sharing details of new products and services with visitors to our site. Segment Connections is a service we and many “digital-native” businesses use to [capture and route all of our customer and visitor interactions](https://rittmananalytics.com/blog/2019/11/6/e058gepqwiyyx4mop3eb06mn9ckjkp) to the downstream marketing, customer service and analytics applications we use to run our operations. Segment Personas is a Customer Data Platform that merges each visitors’ cross-channel behaviour into a consolidated profile that we [use to create a 360-degree profile](https://rittmananalytics.com/blog/2019/12/15/segmentcdpemailtracking) of all of our clients’ interactions with our business. Intercom’s integration with our Segment Personas’ customer data platform goes one steps further and provides Intercom with a much richer and complete 360-degree view of our clients’ previous payments, preferences and new opportunities we might be in-discussion with them on, making those interactions more relevant, more efficient and more likely to result in a successful client outcome. So now, when clients log in to our site and we know who they are, we can make use of Intercom’s integration with Segment Connections and Segment Personas to provide data on what pages they’ve been looking at on the site, the status of their last payment and data on the preferences and interests they’ve shared with us before so that we can make the conversation more relevant to those interests. ## Rittman Analytics Marketing Automation Quickstart Poweredby Segment Personas Rittman Analytics helps businesses unlock insights hidden within their data, drive customer acquisition and increase operational efficiency using modern data analytics tools and our agile, remotely-delivered consulting services.  We’re an authorised Segment Implementation Partner that have enabled businesses in the UK, Europe and North America connect their customer digital touchpoint and centralise their customer data into customer data platforms powered by Segment Personas. Now we can help your business start automating your customer interactions using insights delivered by Segment Personas with our new Marketing Automation Quickstart package: - For Segment customers with an existing investment in a Personas Customer Data Platform (*or we can setup Segment for you as a separate engagement beforehand*) - Activate up to three PoC marketing automation channels e.g. Salesforce Marketing Cloud, Salesforce CRM, Intercom, Hubspot, Drip, Planhat - Delivered over four two-week duration delivery sprints - Delivered for a fixed-price of £16,000 (exclusive of tax and any software service license fees) https://calendly.com/markrittman/initial-discovery-call-with-mark-rittman” target=”\_blank” class=”sqs-block-button-element–medium sqs-block-button-element” style=”background: #313131; color: #fff; padding: 8px 12px; border: 0px; -webkit-box-shadow: 0 -2px 0 rgba(0,0,0,0.15) inset;-moz-box-shadow: 0 -2px 0 rgba(0,0,0,0.15) inset;box-shadow: 0 -2px 0 rgba(0,0,0,0.15) inset;border-radius: 4px; text-decoration: none; display: inline-block;”\>Contact Us[https://embed.acuityscheduling.com/embed/button/17819176.js](https://embed.acuityscheduling.com/embed/button/17819176.js) --- # News on our Marketing Automation Quickstart Package for Segment and Intercom Canonical: https://rittmananalytics.com/blog/2020/07/16/2020-7-16-connecting-intercom-to-segment-personas-for-more-relevant-and-cost-effective-customer-services-agents URL: https://rittmananalytics.com/blog/2020/07/16/2020-7-16-connecting-intercom-to-segment-personas-for-more-relevant-and-cost-effective-customer-services-agents Description: Two trends are emerging clearly from the events of 2020 and the COVID-19 global pandemic; businesses and their customers are moving online, and those businesses need to find, market to… Author: Mark Rittman Published: 2020-07-16T23:58:21+00:00 Modified: 2020-07-16T23:58:21+00:00 --- Two trends are emerging clearly from the events of 2020 and the COVID-19 global pandemic; businesses and their customers are moving online, and those businesses need to find, market to and retain their customers in the most cost-efficient and effective way possible. ## Automate your Customer Service Interactions Intercom is an in-app “chatbot” messaging service we and many other online businesses use to automate the process of qualifying sales leads, answering support questions and sharing details of new products and services with visitors to our site. Segment Connections is a service we and many “digital-native” businesses use to [capture and route all of our customer and visitor interactions](https://rittmananalytics.com/blog/2019/11/6/e058gepqwiyyx4mop3eb06mn9ckjkp) to the downstream marketing, customer service and analytics applications we use to run our operations. Segment Personas is a Customer Data Platform that merges each visitors’ cross-channel behaviour into a consolidated profile that we [use to create a 360-degree profile](https://rittmananalytics.com/blog/2019/12/15/segmentcdpemailtracking) of all of our clients’ interactions with our business. Intercom’s integration with our Segment Personas’ customer data platform goes one steps further and provides Intercom with a much richer and complete 360-degree view of our clients’ previous payments, preferences and new opportunities we might be in-discussion with them on, making those interactions more relevant, more efficient and more likely to result in a successful client outcome. So now, when clients log in to our site and we know who they are, we can make use of Intercom’s integration with Segment Connections and Segment Personas to provide data on what pages they’ve been looking at on the site, the status of their last payment and data on the preferences and interests they’ve shared with us before so that we can make the conversation more relevant to those interests. ## Rittman Analytics Marketing Automation Quickstart Poweredby Segment Personas Rittman Analytics helps businesses unlock insights hidden within their data, drive customer acquisition and increase operational efficiency using modern data analytics tools and our agile, remotely-delivered consulting services.  We’re an authorised Segment Implementation Partner that have enabled businesses in the UK, Europe and North America connect their customer digital touchpoint and centralise their customer data into customer data platforms powered by Segment Personas. Now we can help your business start automating your customer interactions using insights delivered by Segment Personas with our new Marketing Automation Quickstart package: - For Segment customers with an existing investment in a Personas Customer Data Platform (*or we can setup Segment for you as a separate engagement beforehand*) - Activate up to three PoC marketing automation channels e.g. Salesforce Marketing Cloud, Salesforce CRM, Intercom, Hubspot, Drip, Planhat - Delivered over four two-week duration delivery sprints - Delivered for a fixed-price of £16,000 (exclusive of tax and any software service license fees) https://calendly.com/markrittman/initial-discovery-call-with-mark-rittman” target=”\_blank” class=”sqs-block-button-element–medium sqs-block-button-element” style=”background: #313131; color: #fff; padding: 8px 12px; border: 0px; -webkit-box-shadow: 0 -2px 0 rgba(0,0,0,0.15) inset;-moz-box-shadow: 0 -2px 0 rgba(0,0,0,0.15) inset;box-shadow: 0 -2px 0 rgba(0,0,0,0.15) inset;border-radius: 4px; text-decoration: none; display: inline-block;”\>Contact Us[https://embed.acuityscheduling.com/embed/button/17819176.js](https://embed.acuityscheduling.com/embed/button/17819176.js) --- # Drill to Detail Ep.82 ‘Looker Development, Automated Testing and Spectacles’ with Special Guest Josh Temple Canonical: https://rittmananalytics.com/blog/2020/06/04/drill-to-detail-ep-82-looker-development-automated-testing-and-spectacles-with-special-guest-josh-temple URL: https://rittmananalytics.com/blog/2020/06/04/drill-to-detail-ep-82-looker-development-automated-testing-and-spectacles-with-special-guest-josh-temple Description: Over at the Drill to Detail Podcast, Mark Rittman is joined in a new episode by Josh Temple, Analytics Engineer at Spotify to talk about analytics development, automated testing and… Author: Mark Rittman Published: 2020-06-04T12:58:13+01:00 Modified: 2020-06-04T12:58:13+01:00 --- Over at the [Drill to Detail Podcast](https://mjr-analytics.squarespace.com/drilltodetail/2020/6/3/drill-to-detail-ep82-looker-development-automated-testing-and-spectacles-with-special-guest-josh-temple), Mark Rittman is joined in a new episode by Josh Temple, Analytics Engineer at Spotify to talk about analytics development, automated testing and Spectacles, an open-source tool and SaaS service that automatically tests your LookML to ensure Looker always runs smoothly for your users. - [Spectacles product homepage](https://www.spectacles.dev/) and early access program - [Data Tests in Looker](https://docs.looker.com/reference/model-params/test) - [Automated testing in the modern data warehouse](https://medium.com/@josh.temple/automated-testing-in-the-modern-data-warehouse-d5a251a866af) - [#JOIN19 – Using Customized Open-Source Tools With LookML](https://info.looker.com/join-2019/extending-lookml-development-with-customized-toold) - [Continuous Integration and Automated Build Testing with dbtCloud](https://rittmananalytics.com/blog/2019/6/11/continuous-integration-feature-branches-and-automated-build-tests-using-dbtcloud) --- # Column-Level Data Profiling for Google BigQuery Datasets using dbt Canonical: https://rittmananalytics.com/blog/2020/06/03/column-level-data-profiling-for-google-bigquery-datasets-using-dbt URL: https://rittmananalytics.com/blog/2020/06/03/column-level-data-profiling-for-google-bigquery-datasets-using-dbt Description: One of the challenges when centralising data from a new source is how to efficiently audit the data it provides, and one of the most fundamental tasks in a data… Author: Mark Rittman Published: 2020-06-04T00:29:58+01:00 Modified: 2020-06-04T00:29:58+01:00 --- One of the challenges when centralising data from a new source is how to efficiently audit the data it provides, and one of the most fundamental tasks in a data audit is to understand the content and structure of each of those data source tables. The data profiling feature within the RA Warehouse dbt Framework we [blogged about](https://rittmananalytics.com/blog/2020/5/28/introducing-the-ra-warehouse-dbt-framework-how-rittman-analytics-does-data-centralization) and [published to Github](https://github.com/rittmananalytics/ra_data_warehouse) last week makes it easy to capture the following column-level object stats and metadata for an entire dataset (schema) of tables and views in Google BigQuery - Count of nulls, not nulls and percentage null - Whether column is Not Nullable - Count of unique values and percentage unique - Data Type - Min, Max and Average values - Most frequently occurring value, and count of rows containing most frequent value - Whether column is used for partitioning The profiler also generates two recommendations for dbt model tests: - Based on a configurable % threshold (default 90%) whether the column should be considered Not Nullable - Again based on a configurable % threshold (default 90%) whether the column should be considered as a “unique values” column You can then use the recommendations generated by the profiler to indicate where you might want to add UNIQUE or NOT\_NULL tests into the models section of your dbt schema.yml file, with the small percentage (configurable and set by default at 90%) of non-compliant rows then considered as data errors. ### **How Does the Data Profiler Work?** Data Profiling is based around a dbt macro that, for every view or table (“relation”) in a given schema, generates an SQL query that creates a series of SQL query blocks that generate stats for each database object, unions those queries together and then joins the results to another query against the INFORMATION\_SCHEMA.COLUMNS to obtain each object’s metadata. {%- macro profile\_schema(table\_schema) -%}{{ config(schema=’profiles’) }}{% set not\_null\_profile\_threshold\_pct = “.9” %}{% set unique\_profile\_threshold\_pct = “.9″ %}{% set tables = dbt\_utils.get\_relations\_by\_prefix(table\_schema, ”) %}SELECT column\_stats.table\_catalog,column\_stats.table\_schema,column\_stats.table\_name,column\_stats.column\_name,case when column\_metadata.is\_nullable = ‘YES’ then false else true end as is\_not\_nullable\_column,case when column\_stats.pct\_not\_null \> {{ not\_null\_profile\_threshold\_pct }} then true else false end as is\_recommended\_not\_nullable\_column,column\_stats.\_nulls as count\_nulls,column\_stats.\_non\_nulls as count\_not\_nulls,column\_stats.pct\_not\_null as pct\_not\_null,column\_stats.table\_rows,column\_stats.count\_distinct\_values,column\_stats.pct\_unique,case when column\_stats.pct\_unique \>\= {{ unique\_profile\_threshold\_pct }} then true else false end as is\_recommended\_unique\_column,column\_metadata.\* EXCEPT (table\_catalog,table\_schema,table\_name,column\_name,is\_nullable),column\_stats.\* EXCEPT (table\_catalog,table\_schema,table\_name,column\_name,\_nulls,\_non\_nulls,pct\_not\_null,table\_rows,pct\_unique,count\_distinct\_values)FROM({% for table in tables %}SELECT \*FROM(WITH\`table\` AS (SELECT \* FROM {{ table }} ),table\_as\_json AS (SELECT REGEXP\_REPLACE(TO\_JSON\_STRING(t), r’^{|}$’, ”) AS ROW FROM \`table\` AS t ),pairs AS (SELECT REPLACE(column\_name, ‘”‘, ”) AS column\_name, IF (SAFE\_CAST(column\_value AS STRING)=’null’,NULL, column\_value) AS column\_valueFROM table\_as\_json,UNNEST(SPLIT(ROW, ‘,”‘)) AS z,UNNEST(\[SPLIT(z, ‘:’)\[SAFE\_OFFSET(0)\]\]) AS column\_name,UNNEST(\[SPLIT(z, ‘:’)\[SAFE\_OFFSET(1)\]\]) AS column\_value ),profile AS (SELECTsplit(replace(‘{{ table }}’,’\`’,”),’.’ )\[safe\_offset(0)\] as table\_catalog,split(replace(‘{{ table }}’,’\`’,”),’.’ )\[safe\_offset(1)\] as table\_schema,split(replace(‘{{ table }}’,’\`’,”),’.’ )\[safe\_offset(2)\] as table\_name,column\_name,COUNT(\*) AS table\_rows,COUNT(DISTINCT column\_value) AS count\_distinct\_values,safe\_divide(COUNT(DISTINCT column\_value),COUNT(\*)) AS pct\_unique,COUNTIF(column\_value IS NULL) AS \_nulls,COUNTIF(column\_value IS NOT NULL) AS \_non\_nulls,COUNTIF(column\_value IS NOT NULL) / COUNT(\*) AS pct\_not\_null,min(column\_value) as \_min\_value,max(column\_value) as \_max\_value,avg(SAFE\_CAST(column\_value AS numeric)) as \_avg\_value,APPROX\_TOP\_COUNT(column\_value, 1)\[OFFSET(0)\] AS \_most\_frequent\_value,MIN(LENGTH(SAFE\_CAST(column\_value AS STRING))) AS \_min\_length,MAX(LENGTH(SAFE\_CAST(column\_value AS STRING))) AS \_max\_length,ROUND(AVG(LENGTH(SAFE\_CAST(column\_value AS STRING)))) AS \_avr\_lengthFROMpairsWHEREcolumn\_name < \> ”AND column\_name NOT LIKE ‘%-%’GROUP BYcolumn\_nameORDER BYcolumn\_name)SELECT\*FROMprofile){%- if not loop.last %}UNION ALL{%- endif %}{% endfor %}) column\_statsLEFT OUTER JOIN(SELECT\* EXCEPT(is\_generated,generation\_expression,is\_stored,is\_updatable)FROM{{ table\_schema }}.INFORMATION\_SCHEMA.COLUMNS) column\_metadataON column\_stats.table\_catalog = column\_metadata.table\_catalogAND column\_stats.table\_schema = column\_metadata.table\_schemaAND column\_stats.table\_name = column\_metadata.table\_nameAND column\_stats.column\_name = column\_metadata.column\_name{%- endmacro -%} Then, within each data source adapter (folder within a dbt git repo) you will find a model definition such as this one for the Asana Projects source : {% if not var(“enable\_asana\_projects\_source”) %}{{config(enabled=false)}}{% endif %}{% if var(“etl”) == ‘fivetran’ %}{{ profile\_schema(var(‘fivetran\_schema’)) }}{% elif var(“etl”) == ‘stitch’ %}{{ profile\_schema(var(‘stitch\_schema’)) }}{% endif %} These models when run will automatically create views within the the “profile” dataset (e.g. analytics\_profile) that you can use to audit and profile the data from newly-enabled data source adapters (note that you will need to create corresponding model files yourself for any new, custom data source adapters). There is also a “profile\_wh\_tables.sql” model within the /models/utils folder that runs the following jinja code: {{ profile\_schema(target.schema) }} that automatically profiles all of the fact and dimension tables in the warehouse at the end of dbt processing. See the [RA Warehouse dbt Framework](https://github.com/rittmananalytics/ra_data_warehouse) on Github for more details, and our [Data Centralisation consulting service](https://rittmananalytics.com/data-centralisation) if you’d like us to implement dbt or a centralized SaaS data warehouse at your organization. --- # Column-Level Data Profiling for Google BigQuery Datasets using dbt Canonical: https://rittmananalytics.com/blog/2020/06/03/2020-6-3-column-level-data-profiling-for-google-bigquery-datasets-using-dbt URL: https://rittmananalytics.com/blog/2020/06/03/2020-6-3-column-level-data-profiling-for-google-bigquery-datasets-using-dbt Description: One of the challenges when centralising data from a new source is how to efficiently audit the data it provides, and one of the most fundamental tasks in a data… Author: Mark Rittman Published: 2020-06-03T23:29:58+00:00 Modified: 2020-06-03T23:29:58+00:00 --- One of the challenges when centralising data from a new source is how to efficiently audit the data it provides, and one of the most fundamental tasks in a data audit is to understand the content and structure of each of those data source tables. The data profiling feature within the RA Warehouse dbt Framework we [blogged about](https://rittmananalytics.com/blog/2020/5/28/introducing-the-ra-warehouse-dbt-framework-how-rittman-analytics-does-data-centralization) and [published to Github](https://github.com/rittmananalytics/ra_data_warehouse) last week makes it easy to capture the following column-level object stats and metadata for an entire dataset (schema) of tables and views in Google BigQuery - Count of nulls, not nulls and percentage null - Whether column is Not Nullable - Count of unique values and percentage unique - Data Type - Min, Max and Average values - Most frequently occurring value, and count of rows containing most frequent value - Whether column is used for partitioning The profiler also generates two recommendations for dbt model tests: - Based on a configurable % threshold (default 90%) whether the column should be considered Not Nullable - Again based on a configurable % threshold (default 90%) whether the column should be considered as a “unique values” column You can then use the recommendations generated by the profiler to indicate where you might want to add UNIQUE or NOT\_NULL tests into the models section of your dbt schema.yml file, with the small percentage (configurable and set by default at 90%) of non-compliant rows then considered as data errors. ### **How Does the Data Profiler Work?** Data Profiling is based around a dbt macro that, for every view or table (“relation”) in a given schema, generates an SQL query that creates a series of SQL query blocks that generate stats for each database object, unions those queries together and then joins the results to another query against the INFORMATION\_SCHEMA.COLUMNS to obtain each object’s metadata. {%- macro profile\_schema(table\_schema) -%}{{ config(schema=’profiles’) }}{% set not\_null\_profile\_threshold\_pct = “.9” %}{% set unique\_profile\_threshold\_pct = “.9″ %}{% set tables = dbt\_utils.get\_relations\_by\_prefix(table\_schema, ”) %}SELECT column\_stats.table\_catalog,column\_stats.table\_schema,column\_stats.table\_name,column\_stats.column\_name,case when column\_metadata.is\_nullable = ‘YES’ then false else true end as is\_not\_nullable\_column,case when column\_stats.pct\_not\_null \> {{ not\_null\_profile\_threshold\_pct }} then true else false end as is\_recommended\_not\_nullable\_column,column\_stats.\_nulls as count\_nulls,column\_stats.\_non\_nulls as count\_not\_nulls,column\_stats.pct\_not\_null as pct\_not\_null,column\_stats.table\_rows,column\_stats.count\_distinct\_values,column\_stats.pct\_unique,case when column\_stats.pct\_unique \>\= {{ unique\_profile\_threshold\_pct }} then true else false end as is\_recommended\_unique\_column,column\_metadata.\* EXCEPT (table\_catalog,table\_schema,table\_name,column\_name,is\_nullable),column\_stats.\* EXCEPT (table\_catalog,table\_schema,table\_name,column\_name,\_nulls,\_non\_nulls,pct\_not\_null,table\_rows,pct\_unique,count\_distinct\_values)FROM({% for table in tables %}SELECT \*FROM(WITH\`table\` AS (SELECT \* FROM {{ table }} ),table\_as\_json AS (SELECT REGEXP\_REPLACE(TO\_JSON\_STRING(t), r’^{|}$’, ”) AS ROW FROM \`table\` AS t ),pairs AS (SELECT REPLACE(column\_name, ‘”‘, ”) AS column\_name, IF (SAFE\_CAST(column\_value AS STRING)=’null’,NULL, column\_value) AS column\_valueFROM table\_as\_json,UNNEST(SPLIT(ROW, ‘,”‘)) AS z,UNNEST(\[SPLIT(z, ‘:’)\[SAFE\_OFFSET(0)\]\]) AS column\_name,UNNEST(\[SPLIT(z, ‘:’)\[SAFE\_OFFSET(1)\]\]) AS column\_value ),profile AS (SELECTsplit(replace(‘{{ table }}’,’\`’,”),’.’ )\[safe\_offset(0)\] as table\_catalog,split(replace(‘{{ table }}’,’\`’,”),’.’ )\[safe\_offset(1)\] as table\_schema,split(replace(‘{{ table }}’,’\`’,”),’.’ )\[safe\_offset(2)\] as table\_name,column\_name,COUNT(\*) AS table\_rows,COUNT(DISTINCT column\_value) AS count\_distinct\_values,safe\_divide(COUNT(DISTINCT column\_value),COUNT(\*)) AS pct\_unique,COUNTIF(column\_value IS NULL) AS \_nulls,COUNTIF(column\_value IS NOT NULL) AS \_non\_nulls,COUNTIF(column\_value IS NOT NULL) / COUNT(\*) AS pct\_not\_null,min(column\_value) as \_min\_value,max(column\_value) as \_max\_value,avg(SAFE\_CAST(column\_value AS numeric)) as \_avg\_value,APPROX\_TOP\_COUNT(column\_value, 1)\[OFFSET(0)\] AS \_most\_frequent\_value,MIN(LENGTH(SAFE\_CAST(column\_value AS STRING))) AS \_min\_length,MAX(LENGTH(SAFE\_CAST(column\_value AS STRING))) AS \_max\_length,ROUND(AVG(LENGTH(SAFE\_CAST(column\_value AS STRING)))) AS \_avr\_lengthFROMpairsWHEREcolumn\_name < \> ”AND column\_name NOT LIKE ‘%-%’GROUP BYcolumn\_nameORDER BYcolumn\_name)SELECT\*FROMprofile){%- if not loop.last %}UNION ALL{%- endif %}{% endfor %}) column\_statsLEFT OUTER JOIN(SELECT\* EXCEPT(is\_generated,generation\_expression,is\_stored,is\_updatable)FROM{{ table\_schema }}.INFORMATION\_SCHEMA.COLUMNS) column\_metadataON column\_stats.table\_catalog = column\_metadata.table\_catalogAND column\_stats.table\_schema = column\_metadata.table\_schemaAND column\_stats.table\_name = column\_metadata.table\_nameAND column\_stats.column\_name = column\_metadata.column\_name{%- endmacro -%} Then, within each data source adapter (folder within a dbt git repo) you will find a model definition such as this one for the Asana Projects source : {% if not var(“enable\_asana\_projects\_source”) %}{{config(enabled=false)}}{% endif %}{% if var(“etl”) == ‘fivetran’ %}{{ profile\_schema(var(‘fivetran\_schema’)) }}{% elif var(“etl”) == ‘stitch’ %}{{ profile\_schema(var(‘stitch\_schema’)) }}{% endif %} These models when run will automatically create views within the the “profile” dataset (e.g. analytics\_profile) that you can use to audit and profile the data from newly-enabled data source adapters (note that you will need to create corresponding model files yourself for any new, custom data source adapters). There is also a “profile\_wh\_tables.sql” model within the /models/utils folder that runs the following jinja code: {{ profile\_schema(target.schema) }} that automatically profiles all of the fact and dimension tables in the warehouse at the end of dbt processing. See the [RA Warehouse dbt Framework](https://github.com/rittmananalytics/ra_data_warehouse) on Github for more details, and our [Data Centralisation consulting service](https://rittmananalytics.com/data-centralisation) if you’d like us to implement dbt or a centralized SaaS data warehouse at your organization. --- # Column-Level Data Profiling for Google BigQuery Datasets — Rittman Analytics Canonical: https://rittmananalytics.com/blog/2020/06/03/column-level-data-profiling-for-google-bigquery-datasets-rittman-analytics URL: https://rittmananalytics.com/blog/2020/06/03/column-level-data-profiling-for-google-bigquery-datasets-rittman-analytics Description: Column-Level Data Profiling for Google BigQuery Datasets using dbt Modern Analytics StackTips and Techniques 4 Jun Written By Mark Rittman One of the challenges when centralising data from a new… Author: Mark Rittman Published: 2020-06-03T01:00:00+01:00 Modified: 2020-06-03T01:00:00+01:00 --- ![Column-Level Data Profiling for Google BigQuery Datasets — Rittman Analytics](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2020%2F06%2F1752872543183-ivc23a.png%3Ffit%3D1865%252C746%26ssl%3D1) # Column-Level Data Profiling for Google BigQuery Datasets using dbt [Modern Analytics Stack](https://rittmananalytics.com/blog/category/Modern+Analytics+Stack)[Tips and Techniques](https://rittmananalytics.com/blog/category/Tips+and+Techniques) 4 Jun Written By [Mark Rittman](https://rittmananalytics.com/blog?author=63c478ac6811232b3153d585) One of the challenges when centralising data from a new source is how to efficiently audit the data it provides, and one of the most fundamental tasks in a data audit is to understand the content and structure of each of those data source tables. The data profiling feature within the RA Warehouse dbt Framework we [blogged about](https://rittmananalytics.com/blog/2020/5/28/introducing-the-ra-warehouse-dbt-framework-how-rittman-analytics-does-data-centralization) and [published to Github](https://github.com/rittmananalytics/ra_data_warehouse) last week makes it easy to capture the following column-level object stats and metadata for an entire dataset (schema) of tables and views in Google BigQuery - Count of nulls, not nulls and percentage null - Whether column is Not Nullable - Count of unique values and percentage unique - Data Type - Min, Max and Average values - Most frequently occurring value, and count of rows containing most frequent value - Whether column is used for partitioning ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872543183-ivc23a.png%3Fresize%3D1865%252C746%26ssl%3D1) data\_profile.png The profiler also generates two recommendations for dbt model tests: - Based on a configurable % threshold (default 90%) whether the column should be considered Not Nullable - Again based on a configurable % threshold (default 90%) whether the column should be considered as a “unique values” column You can then use the recommendations generated by the profiler to indicate where you might want to add UNIQUE or NOT\_NULL tests into the models section of your dbt schema.yml file, with the small percentage (configurable and set by default at 90%) of non-compliant rows then considered as data errors. ### **How Does the Data Profiler Work?** Data Profiling is based around a dbt macro that, for every view or table (“relation”) in a given schema, generates an SQL query that creates a series of SQL query blocks that generate stats for each database object, unions those queries together and then joins the results to another query against the INFORMATION\_SCHEMA.COLUMNS to obtain each object’s metadata. {%- macro profile\_schema(table\_schema) -%}{{ config(schema=’profiles’) }}{% set not\_null\_profile\_threshold\_pct = “.9” %}{% set unique\_profile\_threshold\_pct = “.9″ %}{% set tables = dbt\_utils.get\_relations\_by\_prefix(table\_schema, ”) %}SELECT column\_stats.table\_catalog,column\_stats.table\_schema,column\_stats.table\_name,column\_stats.column\_name,case when column\_metadata.is\_nullable = ‘YES’ then false else true end as is\_not\_nullable\_column,case when column\_stats.pct\_not\_null \> {{ not\_null\_profile\_threshold\_pct }} then true else false end as is\_recommended\_not\_nullable\_column,column\_stats.\_nulls as count\_nulls,column\_stats.\_non\_nulls as count\_not\_nulls,column\_stats.pct\_not\_null as pct\_not\_null,column\_stats.table\_rows,column\_stats.count\_distinct\_values,column\_stats.pct\_unique,case when column\_stats.pct\_unique \>\= {{ unique\_profile\_threshold\_pct }} then true else false end as is\_recommended\_unique\_column,column\_metadata.\* EXCEPT (table\_catalog,table\_schema,table\_name,column\_name,is\_nullable),column\_stats.\* EXCEPT (table\_catalog,table\_schema,table\_name,column\_name,\_nulls,\_non\_nulls,pct\_not\_null,table\_rows,pct\_unique,count\_distinct\_values)FROM({% for table in tables %}SELECT \*FROM(WITH\`table\` AS (SELECT \* FROM {{ table }} ),table\_as\_json AS (SELECT REGEXP\_REPLACE(TO\_JSON\_STRING(t), r’^{|}$’, ”) AS ROW FROM \`table\` AS t ),pairs AS (SELECT REPLACE(column\_name, ‘”‘, ”) AS column\_name, IF (SAFE\_CAST(column\_value AS STRING)=’null’,NULL, column\_value) AS column\_valueFROM table\_as\_json,UNNEST(SPLIT(ROW, ‘,”‘)) AS z,UNNEST(\[SPLIT(z, ‘:’)\[SAFE\_OFFSET(0)\]\]) AS column\_name,UNNEST(\[SPLIT(z, ‘:’)\[SAFE\_OFFSET(1)\]\]) AS column\_value ),profile AS (SELECTsplit(replace(‘{{ table }}’,’\`’,”),’.’ )\[safe\_offset(0)\] as table\_catalog,split(replace(‘{{ table }}’,’\`’,”),’.’ )\[safe\_offset(1)\] as table\_schema,split(replace(‘{{ table }}’,’\`’,”),’.’ )\[safe\_offset(2)\] as table\_name,column\_name,COUNT(\*) AS table\_rows,COUNT(DISTINCT column\_value) AS count\_distinct\_values,safe\_divide(COUNT(DISTINCT column\_value),COUNT(\*)) AS pct\_unique,COUNTIF(column\_value IS NULL) AS \_nulls,COUNTIF(column\_value IS NOT NULL) AS \_non\_nulls,COUNTIF(column\_value IS NOT NULL) / COUNT(\*) AS pct\_not\_null,min(column\_value) as \_min\_value,max(column\_value) as \_max\_value,avg(SAFE\_CAST(column\_value AS numeric)) as \_avg\_value,APPROX\_TOP\_COUNT(column\_value, 1)\[OFFSET(0)\] AS \_most\_frequent\_value,MIN(LENGTH(SAFE\_CAST(column\_value AS STRING))) AS \_min\_length,MAX(LENGTH(SAFE\_CAST(column\_value AS STRING))) AS \_max\_length,ROUND(AVG(LENGTH(SAFE\_CAST(column\_value AS STRING)))) AS \_avr\_lengthFROMpairsWHEREcolumn\_name < \> ”AND column\_name NOT LIKE ‘%-%’GROUP BYcolumn\_nameORDER BYcolumn\_name)SELECT\*FROMprofile){%- if not loop.last %}UNION ALL{%- endif %}{% endfor %}) column\_statsLEFT OUTER JOIN(SELECT\* EXCEPT(is\_generated,generation\_expression,is\_stored,is\_updatable)FROM{{ table\_schema }}.INFORMATION\_SCHEMA.COLUMNS) column\_metadataON column\_stats.table\_catalog = column\_metadata.table\_catalogAND column\_stats.table\_schema = column\_metadata.table\_schemaAND column\_stats.table\_name = column\_metadata.table\_nameAND column\_stats.column\_name = column\_metadata.column\_name{%- endmacro -%} Then, within each data source adapter (folder within a dbt git repo) you will find a model definition such as this one for the Asana Projects source : {% if not var(“enable\_asana\_projects\_source”) %}{{config(enabled=false)}}{% endif %}{% if var(“etl”) == ‘fivetran’ %}{{ profile\_schema(var(‘fivetran\_schema’)) }}{% elif var(“etl”) == ‘stitch’ %}{{ profile\_schema(var(‘stitch\_schema’)) }}{% endif %} These models when run will automatically create views within the the “profile” dataset (e.g. analytics\_profile) that you can use to audit and profile the data from newly-enabled data source adapters (note that you will need to create corresponding model files yourself for any new, custom data source adapters). There is also a “profile\_wh\_tables.sql” model within the /models/utils folder that runs the following jinja code: {{ profile\_schema(target.schema) }} that automatically profiles all of the fact and dimension tables in the warehouse at the end of dbt processing. See the [RA Warehouse dbt Framework](https://github.com/rittmananalytics/ra_data_warehouse) on Github for more details, and our [Data Centralisation consulting service](https://rittmananalytics.com/data-centralisation) if you’d like us to implement dbt or a centralized SaaS data warehouse at your organization. [Data Engineering](https://rittmananalytics.com/blog/tag/Data+Engineering)[dbt](https://rittmananalytics.com/blog/tag/dbt) [Mark Rittman](https://rittmananalytics.com/blog?author=63c478ac6811232b3153d585) --- # How Rittman Analytics Centralizes SaaS Data Sources using dbt and Google BigQuery Canonical: https://rittmananalytics.com/blog/2020/05/28/how-rittman-analytics-centralizes-saas-data-sources-using-dbt-and-google-bigquery URL: https://rittmananalytics.com/blog/2020/05/28/how-rittman-analytics-centralizes-saas-data-sources-using-dbt-and-google-bigquery Description: Around this time last year I put together a blog post titled “How Rittman Analytics Does Analytics” in which I went through the tools and delivery approach we used when… Author: Mark Rittman Published: 2020-05-28T12:00:00+01:00 Modified: 2026-07-02T00:31:20+01:00 --- ![How Rittman Analytics Centralizes SaaS Data Sources using dbt and Google BigQuery](https://cdn-images-1.medium.com/max/800/0*MInPMoRzIhU93gPX) Around this time last year I put together a blog post titled [“How Rittman Analytics Does Analytics”](https://rittmananalytics.com/blog/2019/5/7/how-rittman-analytics) in which I went through the tools and delivery approach we used when building analytics solutions for ourselves and our clients using Looker, Google BigQuery, dbt and Stitch. In that initial blog article I focused more on our Looker dashboards and the framework of KPIs we used for our own internal analytics. In this update to that post I’m going to focus this time on the underlying data warehouse layer and how we use dbt to model, transform and load data into that warehouse using our [RA Data Warehouse dbt Framework](https://github.com/rittmananalytics/ra_data_warehouse) we’ve recently published for public access, forking and PRs on Github. ![](https://cdn-images-1.medium.com/max/800/0*MInPMoRzIhU93gPX) So what does this framework contain, and what problems does it solve for our team of analytics engineers working on our client and internal projects? ### What is the RA Warehouse dbt Framework? The RA Warehouse dbt framework is a set of data models, data transformations and data warehouse dimensional models we use for ingesting, combining and restructuring data from multiple source systems into a conformed, Kimball-style data warehouse using - Google BigQuery (Standard SQL) - dbt - Stitch - Fivetran (limited support as of now) - Segment (limited support, but along with Fivetran, on our immediate roadmap to complete ![](https://cdn-images-1.medium.com/max/800/0*fJEKULbHVSe14xD2) For anyone who’s not so familiar with [dbt](https://www.getdbt.com/), it’s an open-source toolkit sponsored by our friends and partners over at [Fishtown Analytics](https://www.fishtownanalytics.com/) that solves the problem of testing, repeatability and modularity of analysts code by bringing the principles of modern software development to the analysts’ workflow. Our data warehousing framework for dbt tries to solve the problem of how to design your warehouse so that your project doesn’t grind to a halt after you integrate your second, third, fourth data source. It’s been our experience that complexity increases exponentially in data centralization projects as you add more sources into the warehouse when you don’t properly plan for things such as - combining identity across multiple systems when none of those systems are the definitive source of that data - deduplicating multiple sources of customer data even from within the same source system, - keeping the velocity and agility of project delivery consistent even as complexity increases - and most importantly. making sure that numbers you deliver through the warehouse are trusted, tested and actually add up properly ### Design Goals The design goals we set ourselves for this framework were therefore: 1. For Rittman Analytics’ team, to provide some standards around how we model and transform various data sources — coding standards and DRY code, house formatting for SQL etc 2. To make it simpler to run data quality tests than to not, by defining these tests in-advance 3. To enable merging of customer, product, contact and other shared entity data with no single authoritative source 4. To pre-create derived analytics measures for individual and combinations of sources 5. To create a means of selecting sources or subject areas (“modules”) and have just those sources/modules loaded (and deployed for a customer) 6. To enable use of either Stitch, Fivetran or Segment as the pipeline technology based on client need 7. To enable loading and integration of custom (customer app database) sources into the warehouse We do this by separating data sources and data transformations out into “adapter” and “integration” dbt models, which together with data sources and “warehouse modules” gives us a multi-layer warehouse architecture like this: ![](https://cdn-images-1.medium.com/max/800/0*RnHcf2NU_Q65xKr8) Data flows through the layers in the form of a dbt DAG (“Directed Acyclic Graph”) of data transformations: ![](https://cdn-images-1.medium.com/max/800/0*Y3ppa4pffi1iKmsL) and populates a conformed, dimensional Kimball-style data warehouse. ![](https://cdn-images-1.medium.com/max/800/0*gznLZPxpCXc_PTFZ) ### How Do We Structure the dbt Project? We’ve used a dbt project structure that looks like this: ├── analysis ├── data <\-- "seed" files used for matching/merging companies, projects etc ├── macros ├── models │ ├── integration <\-- "integration" models used to merge and dedupe models across multiple sources │ ├── sources │ │ ├── stg\_asana\_projects. <\-- "source" models with data-source specific transformations and │ │ ├── stg\_custom\_source\_1 renaming of columns into common formats. Where more than one │ │ ├── stg\_custom\_source\_2 pipeline technology (Stitch, Fivetran etc) is supported, these will │ │ ├── stg\_facebook\_ads contain SQL and jinja code for each pipeline type within the one model │ │ ├── stg\_gcp\_billing\_export with the etl type configurable in the dbt\_project.yml config file │ │ ├── stg\_google\_ads │ │ ├── stg\_harvest\_projects │ │ ├── stg\_hubspot\_crm │ │ ├── stg\_intercom\_messaging │ │ ├── stg\_jira\_projects │ │ ├── stg\_mailchimp\_email │ │ ├── stg\_mixpanel\_events │ │ ├── stg\_segment\_events │ │ ├── stg\_stripe\_payments │ │ ├── stg\_unknown\_values │ │ └── stg\_xero\_accounting │ ├── utils <\-- "utils" models, for example for row count logging │ └── warehouse <\-- "warehouse" models containing fact and dimension tables, │ ├── w\_crm grouped by subject area │ ├── w\_finance │ ├── w\_marketing │ └── w\_projects Each data source adapter loads the same columns in the same order for tables that are common to multiple sources, for example: WITH source AS ( { } ), renamed AS ( SELECT concat('{ }',gid) as user\_id, name as user\_name , email as user\_email , cast(null as boolean) as user\_is\_contractor, case when email like '%@{ }%' then true else false end as user\_is\_staff, cast(null as int64) as user\_weekly\_capacity, cast(null as string) as user\_phone, cast(null as int64) as user\_default\_hourly\_rate, cast(null as int64) as user\_cost\_rate, true as user\_is\_active, cast(null as timestamp) as user\_created\_ts, cast(null as timestamp) as user\_last\_modified\_ts FROM source WHERE name NOT LIKE 'Private User' ) SELECT \* FROM renamed Custom adapters are also provided to provide mappings into these common structures for one-off data sources specific to an implementation, i.e. a custom app database source. ### What SaaS Sources and Warehouse Marts are Supported? Right now, the RA Warehouse Framework supports the following SaaS data sources and pipeline technologies: - Hubspot CRM (Stitch, Fivetran) - Harvest Timesheets (Stitch) - Xero Accounting (Stitch) - Stripe Payments (Stitch) - Asana Projects (Stitch) - Jira Projects (Stitch) - Mailchimp Email Marketing (Stitch) - Segment Events (Segment) - GCP Billing Exports - Google Ads (Stitch) - Facebook Ads (Stitch) - Intercom Messaging (Stitch) - Mixpanel Events (Stitch, Fivetran) - Custom data sources From these sources we currently load and populate the following warehouse “data marts”. - Finance (Invoices, Chart of Accounts, Currencies) - CRM (Deals, Contacts, Companies) - Projects (Timesheet Projects, Timesheet Tasks, Delivery Projects, Delivery Tasks, Timesheets, Users) - Marketing (Email lists, Email sends, Email campaigns, Ad Campaigns, Ad Performance, Web Page Views, Web Sessions) ### Selectable and Configurable Data Sources The particular data sources enabled for a warehouse implementation can be enabled or disabled by the setting of a flag in the dbt\_project.yml configuration file, like this: vars: enable\_harvest\_projects\_source: \[true|false\] enable\_hubspot\_crm\_source: \[true|false\] enable\_asana\_projects\_source: \[true|false\] enable\_jira\_projects\_source: \[true|false\] enable\_stripe\_payments\_source: \[true|false\] enable\_xero\_accounting\_source: \[true|false\] enable\_mailchimp\_email\_source: \[true|false\] enable\_segment\_events\_source: \[true|false\] enable\_google\_ads\_source: \[true|false\] enable\_facebook\_ads\_source: \[true|false\] enable\_intercom\_messaging\_source: \[true|false\] enable\_custom\_source\_1: \[true|false\] enable\_custom\_source\_2: \[true|false\] enable\_mixpanel\_events\_source: \[true|false\] \# warehouse modules enable\_crm\_warehouse: \[true|false\]e enable\_finance\_warehouse: \[true|false\] enable\_projects\_warehouse: \[true|false\] enable\_marketing\_warehouse: \[true|false\] enable\_ads\_warehouse: \[true|false\] enable\_product\_warehouse: \[true|false\] Each data source then comes with its own model-scoped jinja variables that provide the schema and table names for each of the data source tables, in some cases let you switch between Fivetran, Stitch or Segment data pipelines and enable source-specific settings and choices: stg\_hubspot\_crm: vars: id-prefix: hubspot- etl: stitch stitch\_companies\_table: stitch\_hubspot.companies stitch\_contacts\_table: stitch\_hubspot.contacts stitch\_deals\_table: stitch\_hubspot.deals stitch\_owners\_table: stitch\_hubspot.owners stitch\_pipeline\_stages\_table: stitch\_hubspot.pipeline\_stages stitch\_deal\_pipelines\_table: stitch\_hubspot.deal\_pipelines fivetran\_company\_table: fivetran\_hubspot\_euwest2.company fivetran\_contact\_table: fivetran\_hubspot\_euwest2.contact ### Merging and Combining Dimensions Across Multiple Data Sources Customers, contacts, projects and other shared dimensions are automatically created from all data sources, deduplicating by name and merge lookup files using a process that preserves source system keys whilst assigning a unique ID for each customer, contact etc. Each set of model sources for a given dimension provide a unique ID, prefixed with the source name, and another field value (for example, user name) that can be used for deduplicating dimension members downstream. These fields are then initially merged (UNION ALL) together in the integration layer of the warehouse: ![](https://cdn-images-1.medium.com/max/800/0*3558bjwXw1OCqq7Y) An CTE containing an array of source dimension IDs is then created within the int\_ integration view, grouped by the deduplication column (in this example, user name). Any other multivalue columns are similarly-grouped by the deduplication column in further CTEs within the integration models, for example list of email addresses for a user. user\_emails as ( SELECT user\_name, array\_agg(distinct lower(user\_email) ignore nulls) as all\_user\_emails FROM t\_users\_merge\_list group by 1), user\_ids as ( SELECT user\_name, array\_agg(user\_id ignore nulls) as all\_user\_ids FROM t\_users\_merge\_list group by 1) select i.all\_user\_ids, u.\*, e.all\_user\_emails from ( select user\_name, ... FROM t\_users\_merge\_list group by 1) u join user\_emails e on u.user\_name = coalesce(e.user\_name,'Unknown') join user\_ids i on u.user\_name = i.user\_name For dimensions where merging of members by name is not sufficient (for example, company names that cannot be relied on to always be spelt the same across all sources) we can add seed files to map one member to another and then extend the logic of the merge to make use of this merge file. Within BigQuery these arrays of email addresses, user IDs and addresses are stored as nested, repeated groups of columns that maximise storage efficiency within BigQuery and are automatically “unravelled” when brought into Looker. ### Data Profiling, ETL Results Reporting and Other dbt Macros Finally, along with dbt schema and custom schema tests we make extensive use of macros to enable us to report on the outcome of each dbt Cloud job run and profile source and warehouse target tables and views. For example, this macro calculates a set of useful column statistics all views and tables within a given BigQuery dataset: {%- macro profile\_schema(table\_schema) -%} {% set tables = dbt\_utils.get\_relations\_by\_prefix(table\_schema, '') %} SELECT column\_stats.table\_catalog, column\_stats.table\_schema, column\_stats.table\_name, column\_stats.column\_name, case when column\_stats.pct\_null <\= .1 then 'NULL' else 'NOT NULL' end as recommended\_nullable, case when column\_stats.pct\_null \> 0 and column\_stats.pct\_null <\= .1 then false else true end as is\_recommended\_nullable\_compliant, case when column\_stats.pct\_unique \>\= .9 then 'UNIQUE' else 'NOT UNIQUE' end as recommended\_unique\_key, case when column\_stats.pct\_unique \>\= .9 and column\_stats.pct\_unique < 1 then false else true end as is\_recommended\_unique\_key\_compliant, column\_metadata.\* EXCEPT (table\_catalog, table\_schema, table\_name, column\_name), column\_stats.\* EXCEPT (table\_catalog, table\_schema, table\_name, column\_name) FROM ( {% for table in tables %} SELECT \* FROM ( WITH \`table\` AS (SELECT \* FROM { } ), table\_as\_json AS (SELECT REGEXP\_REPLACE(TO\_JSON\_STRING(t), r'^{|}$', '') AS ROW FROM \`table\` AS t ), pairs AS (SELECT REPLACE(column\_name, '"', '') AS column\_name, IF (SAFE\_CAST(column\_value AS STRING)='null',NULL, column\_value) AS column\_value FROM table\_as\_json,UNNEST(SPLIT(ROW, ',"')) AS z,UNNEST(\[SPLIT(z, ':')\[SAFE\_OFFSET(0)\]\]) AS column\_name,UNNEST(\[SPLIT(z, ':')\[SAFE\_OFFSET(1)\]\]) AS column\_value ), profile AS ( SELECT split(replace('{ }','\`',''),'.' )\[safe\_offset(0)\] as table\_catalog, split(replace('{ }','\`',''),'.' )\[safe\_offset(1)\] as table\_schema, split(replace('{ }','\`',''),'.' )\[safe\_offset(2)\] as table\_name, column\_name, COUNT(\*) AS table\_rows, COUNT(DISTINCT column\_value) AS \_distinct\_values, safe\_divide(COUNT(DISTINCT column\_value),COUNT(\*)) AS pct\_unique, COUNTIF(column\_value IS NULL) AS \_nulls, COUNTIF(column\_value IS NOT NULL) AS \_non\_nulls, COUNTIF(column\_value IS NULL) / COUNT(\*) AS pct\_null, min(column\_value) as \_min\_value, max(column\_value) as \_max\_value, avg(SAFE\_CAST(column\_value AS numeric)) as \_avg\_value, APPROX\_TOP\_COUNT(column\_value, 1)\[OFFSET(0)\] AS \_most\_frequent\_value, MIN(LENGTH(SAFE\_CAST(column\_value AS STRING))) AS \_min\_length, MAX(LENGTH(SAFE\_CAST(column\_value AS STRING))) AS \_max\_length, ROUND(AVG(LENGTH(SAFE\_CAST(column\_value AS STRING)))) AS \_avr\_length FROM pairs WHERE column\_name <\> '' AND column\_name NOT LIKE '%-%' GROUP BY column\_name ORDER BY column\_name) SELECT \* FROM profile) {%- if not loop.last %} UNION ALL {%- endif %} {% endfor %} ) column\_stats LEFT OUTER JOIN ( SELECT \* EXCEPT (is\_generated, generation\_expression, is\_stored, is\_updatable) FROM { }.INFORMATION\_SCHEMA.COLUMNS ) column\_metadata ON column\_stats.table\_catalog = column\_metadata.table\_catalog AND column\_stats.table\_schema = column\_metadata.table\_schema AND column\_stats.table\_name = column\_metadata.table\_name AND column\_stats.column\_name = column\_metadata.column\_name {%- endmacro -%} Then, combined with a data visualization in Looker, we can quickly identify those table columns that are nearly unique or not null so that we can focus our time on addressing what are probably data errors in the rows that aren’t unique or contain null values. ![](https://cdn-images-1.medium.com/max/800/0*OMf0WnHn4Oq55k8W) ### What Features and Other Support are on the Product Roadmap? High-priority features we plan to add in the very near future include: - Extending Fivetran support to cover all remaining data source types - Add more support for incremental loading of warehouse tables - Complete test coverage of warehouse and integration layer models Medium-priority are: - Extending Segment support to cover all appropriate remaining data sources - Snowflake as an alternative to Google BigQuery as the warehouse platform - Enrichment plugins, e.g. Clearbit ### Help Build The Missing Layer in the Modern BI Stack Community Contributions are welcome, and in-fact we’re hoping this project might in-time become an open-source “content and data warehousing” layer on-top of dbt, filling in the missing layer in the open-source modern BI stack that we felt was missing until now. ![](https://cdn-images-1.medium.com/max/800/0*mo4xk9cUgvdg5Owh) ### How Do I Find Out More? The [RA Warehouse dbt package](https://github.com/rittmananalytics/ra_data_warehouse) is now up on Github as a public repo, and feel free to email us at [info@rittmananalytics.com](mailto:info@rittmananalytics.com) for more information … or if you’d like us to bring this framework and our data centralization services to your organization. *First published on the Rittman Analytics blog on* [*28th May 2020*](https://mjr-analytics.squarespace.com/blog/2020/5/28/introducing-the-ra-warehouse-dbt-framework-how-rittman-analytics-does-data-centralization)*.* --- # Introducing the RA Warehouse dbt Framework : How Rittman Analytics Does Data Centralization using dbt, Google BigQuery, Stitch and Looker Canonical: https://rittmananalytics.com/blog/2020/05/27/2020-5-27-introducing-the-ra-warehouse-dbt-framework-how-rittman-analytics-does-data-centralization URL: https://rittmananalytics.com/blog/2020/05/27/2020-5-27-introducing-the-ra-warehouse-dbt-framework-how-rittman-analytics-does-data-centralization Description: Around this time last year I put together a blog post titled “How Rittman Analytics Does Analytics” in which I went through the tools and delivery approach we used when… Author: Mark Rittman Published: 2020-05-27T23:08:56+00:00 Modified: 2020-05-27T23:08:56+00:00 --- Around this time last year I put together a blog post titled [“How Rittman Analytics Does Analytics”](https://rittmananalytics.com/blog/2019/5/7/how-rittman-analytics) in which I went through the tools and delivery approach we used when building analytics solutions for ourselves and our clients using Looker, Google BigQuery, dbt and Stitch. In that initial blog article I focused more on our Looker dashboards and the framework of KPIs we used for our own internal analytics. In this update to that post I’m going to focus this time on the underlying data warehouse layer and how we use dbt to model, transform and load data into that warehouse using our [RA Data Warehouse dbt Framework](https://github.com/rittmananalytics/ra_data_warehouse) we’ve recently published for public access, forking and PRs on Github. So what does this framework contain, and what problems does it solve for our team of analytics engineers working on our client and internal projects? ## What is the RA Warehouse dbt Framework? The RA Warehouse dbt framework is a set of data models, data transformations and data warehouse dimensional models we use for ingesting, combining and restructuring data from multiple source systems into a conformed, Kimball-style data warehouse using - Google BigQuery (Standard SQL) - dbt - Stitch - Fivetran (limited support as of now) - Segment (limited support, but along with Fivetran, on our immediate roadmap to complete For anyone who’s not so familiar with [dbt](https://www.getdbt.com/), it’s an open-source toolkit sponsored by our friends and partners over at [Fishtown Analytics](https://www.fishtownanalytics.com/) that solves the problem of testing, repeatability and modularity of analysts code by bringing the principles of modern software development to the analysts’ workflow. Our data warehousing framework for dbt tries to solve the problem of how to design your warehouse so that your project doesn’t grind to a halt after you integrate your second, third, fourth data source. It’s been our experience that complexity increases exponentially in data centralization projects as you add more sources into the warehouse when you don’t properly plan for things such as - combining identity across multiple systems when none of those systems are the definitive source of that data - deduplicating multiple sources of customer data even from within the same source system, - keeping the velocity and agility of project delivery consistent even as complexity increases - and most importantly. making sure that numbers you deliver through the warehouse are trusted, tested and actually add up properly ## Design Goals The design goals we set ourselves for this framework were therefore: 1. For Rittman Analytics’ team, to provide some standards around how we model and transform various data sources – coding standards and DRY code, house formatting for SQL etc 2. To make it simpler to run data quality tests than to not, by defining these tests in-advance 3. To enable merging of customer, product, contact and other shared entity data with no single authoritative source 4. To pre-create derived analytics measures for individual and combinations of sources 5. To create a means of selecting sources or subject areas (“modules”) and have just those sources/modules loaded (and deployed for a customer) 6. To enable use of either Stitch, Fivetran or Segment as the pipeline technology based on client need 7. To enable loading and integration of custom (customer app database) sources into the warehouse We do this by separating data sources and data transformations out into “adapter” and “integration” dbt models, which together with data sources and “warehouse modules” gives us a multi-layer warehouse architecture like this: Data flows through the layers in the form of a dbt DAG (“Directed Acyclic Graph”) of data transformations: and populates a conformed, dimensional Kimball-style data warehouse. ### How Do We Structure the dbt Project? We’ve used a dbt project structure that looks like this: ``` ├── analysis ├── data <-- "seed" files used for matching/merging companies, projects etc ├── macros ├── models │ ├── integration <-- "integration" models used to merge and dedupe models across multiple sources │ ├── sources │ │ ├── stg_asana_projects. <-- "source" models with data-source specific transformations and │ │ ├── stg_custom_source_1 renaming of columns into common formats. Where more than one │ │ ├── stg_custom_source_2 pipeline technology (Stitch, Fivetran etc) is supported, these will │ │ ├── stg_facebook_ads contain SQL and jinja code for each pipeline type within the one model │ │ ├── stg_gcp_billing_export with the etl type configurable in the dbt_project.yml config file │ │ ├── stg_google_ads │ │ ├── stg_harvest_projects │ │ ├── stg_hubspot_crm │ │ ├── stg_intercom_messaging │ │ ├── stg_jira_projects │ │ ├── stg_mailchimp_email │ │ ├── stg_mixpanel_events │ │ ├── stg_segment_events │ │ ├── stg_stripe_payments │ │ ├── stg_unknown_values │ │ └── stg_xero_accounting │ ├── utils <-- "utils" models, for example for row count logging │ └── warehouse <-- "warehouse" models containing fact and dimension tables, │ ├── w_crm grouped by subject area │ ├── w_finance │ ├── w_marketing │ ├──w_projects ``` Each data source adapter loads the same columns in the same order for tables that are common to multiple sources, for example: WITH source AS ({ }),renamed AS (SELECTconcat(‘{ }’,gid) as user\_id,name as user\_name ,email as user\_email ,cast(null as boolean) as user\_is\_contractor,case when email like ‘%@{ }%’ then true else false end as user\_is\_staff,cast(null as int64) as user\_weekly\_capacity,cast(null as string) as user\_phone,cast(null as int64) as user\_default\_hourly\_rate,cast(null as int64) as user\_cost\_rate,true as user\_is\_active,cast(null as timestamp) as user\_created\_ts,cast(null as timestamp) as user\_last\_modified\_tsFROMsourceWHEREname NOT LIKE ‘Private User’)SELECT\*FROMrenamed Custom adapters are also provided to provide mappings into these common structures for one-off data sources specific to an implementation, i.e. a custom app database source. ## What SaaS Sources and Warehouse Marts are Supported? Right now, the RA Warehouse Framework supports the following SaaS data sources and pipeline technologies: - Hubspot CRM (Stitch, Fivetran) - Harvest Timesheets (Stitch) - Xero Accounting (Stitch) - Stripe Payments (Stitch) - Asana Projects (Stitch) - Jira Projects (Stitch) - Mailchimp Email Marketing (Stitch) - Segment Events (Segment) - GCP Billing Exports - Google Ads (Stitch) - Facebook Ads (Stitch) - Intercom Messaging (Stitch) - Mixpanel Events (Stitch, Fivetran) - Custom data sources From these sources we currently load and populate the following warehouse “data marts”. - Finance (Invoices, Chart of Accounts, Currencies) - CRM (Deals, Contacts, Companies) - Projects (Timesheet Projects, Timesheet Tasks, Delivery Projects, Delivery Tasks, Timesheets, Users) - Marketing (Email lists, Email sends, Email campaigns, Ad Campaigns, Ad Performance, Web Page Views, Web Sessions) ### Selectable and Configurable Data Sources The particular data sources enabled for a warehouse implementation can be enabled or disabled by the setting of a flag in the dbt\_project.yml configuration file, like this: vars:enable\_harvest\_projects\_source: \[true|false\]enable\_hubspot\_crm\_source: \[true|false\]enable\_asana\_projects\_source: \[true|false\]enable\_jira\_projects\_source: \[true|false\]enable\_stripe\_payments\_source: \[true|false\]enable\_xero\_accounting\_source: \[true|false\]enable\_mailchimp\_email\_source: \[true|false\]enable\_segment\_events\_source: \[true|false\]enable\_google\_ads\_source: \[true|false\]enable\_facebook\_ads\_source: \[true|false\]enable\_intercom\_messaging\_source: \[true|false\]enable\_custom\_source\_1: \[true|false\]enable\_custom\_source\_2: \[true|false\]enable\_mixpanel\_events\_source: \[true|false\]# warehouse modulesenable\_crm\_warehouse: \[true|false\]eenable\_finance\_warehouse: \[true|false\]enable\_projects\_warehouse: \[true|false\]enable\_marketing\_warehouse: \[true|false\]enable\_ads\_warehouse: \[true|false\]enable\_product\_warehouse: \[true|false\] Each data source then comes with its own model-scoped jinja variables that provide the schema and table names for each of the data source tables, in some cases let you switch between Fivetran, Stitch or Segment data pipelines and enable source-specific settings and choices: stg\_hubspot\_crm:vars:id-prefix: hubspot-etl: stitchstitch\_companies\_table: stitch\_hubspot.companiesstitch\_contacts\_table: stitch\_hubspot.contactsstitch\_deals\_table: stitch\_hubspot.dealsstitch\_owners\_table: stitch\_hubspot.ownersstitch\_pipeline\_stages\_table: stitch\_hubspot.pipeline\_stagesstitch\_deal\_pipelines\_table: stitch\_hubspot.deal\_pipelinesfivetran\_company\_table: fivetran\_hubspot\_euwest2.companyfivetran\_contact\_table: fivetran\_hubspot\_euwest2.contact ## Merging and Combining Dimensions Across Multiple Data Sources Customers, contacts, projects and other shared dimensions are automatically created from all data sources, deduplicating by name and merge lookup files using a process that preserves source system keys whilst assigning a unique ID for each customer, contact etc. Each set of model sources for a given dimension provide a unique ID, prefixed with the source name, and another field value (for example, user name) that can be used for deduplicating dimension members downstream. These fields are then initially merged (UNION ALL) together in the integration layer of the warehouse: An CTE containing an array of source dimension IDs is then created within the int\_ integration view, grouped by the deduplication column (in this example, user name). Any other multivalue columns are similarly-grouped by the deduplication column in further CTEs within the integration models, for example list of email addresses for a user. user\_emails as (SELECT user\_name, array\_agg(distinct lower(user\_email) ignore nulls) as all\_user\_emailsFROM t\_users\_merge\_listgroup by 1),user\_ids as (SELECT user\_name, array\_agg(user\_id ignore nulls) as all\_user\_idsFROM t\_users\_merge\_listgroup by 1)select i.all\_user\_ids,u.\*,e.all\_user\_emailsfrom (select user\_name,…FROM t\_users\_merge\_listgroup by 1) ujoin user\_emails e on u.user\_name = coalesce(e.user\_name,’Unknown’)join user\_ids i on u.user\_name = i.user\_name For dimensions where merging of members by name is not sufficient (for example, company names that cannot be relied on to always be spelt the same across all sources) we can add seed files to map one member to another and then extend the logic of the merge to make use of this merge file. Within BigQuery these arrays of email addresses, user IDs and addresses are stored as nested, repeated groups of columns that maximise storage efficiency within BigQuery and are automatically “unravelled” when brought into Looker. ### **Data Profiling, ETL Results Reporting and Other dbt Macros** Finally, along with dbt schema and custom schema tests we make extensive use of macros to enable us to report on the outcome of each dbt Cloud job run and profile source and warehouse target tables and views. For example, this macro calculates a set of useful column statistics all views and tables within a given BigQuery dataset: {%- macro profile\_schema(table\_schema) -%}{% set tables = dbt\_utils.get\_relations\_by\_prefix(table\_schema, ”) %}SELECT column\_stats.table\_catalog,column\_stats.table\_schema,column\_stats.table\_name,column\_stats.column\_name,case when column\_stats.pct\_null <\= .1 then ‘NULL’ else ‘NOT NULL’ end as recommended\_nullable,case when column\_stats.pct\_null \> 0 and column\_stats.pct\_null <\= .1 then false else true end as is\_recommended\_nullable\_compliant,case when column\_stats.pct\_unique \>\= .9 then ‘UNIQUE’ else ‘NOT UNIQUE’ end as recommended\_unique\_key,case when column\_stats.pct\_unique \>\= .9 and column\_stats.pct\_unique < 1 then false else true end as is\_recommended\_unique\_key\_compliant,column\_metadata.\* EXCEPT (table\_catalog,table\_schema,table\_name,column\_name),column\_stats.\* EXCEPT (table\_catalog,table\_schema,table\_name,column\_name)FROM({% for table in tables %}SELECT \*FROM(WITH\`table\` AS (SELECT \* FROM { } ),table\_as\_json AS (SELECT REGEXP\_REPLACE(TO\_JSON\_STRING(t), r’^{|}$’, ”) AS ROW FROM \`table\` AS t ),pairs AS (SELECT REPLACE(column\_name, ‘”‘, ”) AS column\_name, IF (SAFE\_CAST(column\_value AS STRING)=’null’,NULL, column\_value) AS column\_valueFROM table\_as\_json,UNNEST(SPLIT(ROW, ‘,”‘)) AS z,UNNEST(\[SPLIT(z, ‘:’)\[SAFE\_OFFSET(0)\]\]) AS column\_name,UNNEST(\[SPLIT(z, ‘:’)\[SAFE\_OFFSET(1)\]\]) AS column\_value ),profile AS (SELECTsplit(replace(‘{ }’,’\`’,”),’.’ )\[safe\_offset(0)\] as table\_catalog,split(replace(‘{ }’,’\`’,”),’.’ )\[safe\_offset(1)\] as table\_schema,split(replace(‘{ }’,’\`’,”),’.’ )\[safe\_offset(2)\] as table\_name,column\_name,COUNT(\*) AS table\_rows,COUNT(DISTINCT column\_value) AS \_distinct\_values,safe\_divide(COUNT(DISTINCT column\_value),COUNT(\*)) AS pct\_unique,COUNTIF(column\_value IS NULL) AS \_nulls,COUNTIF(column\_value IS NOT NULL) AS \_non\_nulls,COUNTIF(column\_value IS NULL) / COUNT(\*) AS pct\_null,min(column\_value) as \_min\_value,max(column\_value) as \_max\_value,avg(SAFE\_CAST(column\_value AS numeric)) as \_avg\_value,APPROX\_TOP\_COUNT(column\_value, 1)\[OFFSET(0)\] AS \_most\_frequent\_value,MIN(LENGTH(SAFE\_CAST(column\_value AS STRING))) AS \_min\_length,MAX(LENGTH(SAFE\_CAST(column\_value AS STRING))) AS \_max\_length,ROUND(AVG(LENGTH(SAFE\_CAST(column\_value AS STRING)))) AS \_avr\_lengthFROMpairsWHEREcolumn\_name <\> ”AND column\_name NOT LIKE ‘%-%’GROUP BYcolumn\_nameORDER BYcolumn\_name)SELECT\*FROMprofile){%- if not loop.last %}UNION ALL{%- endif %}{% endfor %}) column\_statsLEFT OUTER JOIN(SELECT\* EXCEPT(is\_generated,generation\_expression,is\_stored,is\_updatable)FROM{ }.INFORMATION\_SCHEMA.COLUMNS) column\_metadataON column\_stats.table\_catalog = column\_metadata.table\_catalogAND column\_stats.table\_schema = column\_metadata.table\_schemaAND column\_stats.table\_name = column\_metadata.table\_nameAND column\_stats.column\_name = column\_metadata.column\_name{%- endmacro -%} Then, combined with a data visualization in Looker, we can quickly identify those table columns that are nearly unique or not null so that we can focus our time on addressing what are probably data errors in the rows that aren’t unique or contain null values. ## **What Features and Other Support are on the Product Roadmap?** High-priority features we plan to add in the very near future include: - Extending Fivetran support to cover all remaining data source types - Add more support for incremental loading of warehouse tables - Complete test coverage of warehouse and integration layer models Medium-priority are: - Extending Segment support to cover all appropriate remaining data sources - Snowflake as an alternative to Google BigQuery as the warehouse platform - Enrichment plugins, e.g. Clearbit ### **Help Build The Missing Layer in the Modern BI Stack** Community Contributions are welcome, and in-fact we’re hoping this project might in-time become an open-source “content and data warehousing” layer on-top of dbt, filling in the missing layer in the open-source modern BI stack that we felt was missing until now. ## **How Do I Find Out More?** The [RA Warehouse dbt package](https://github.com/rittmananalytics/ra_data_warehouse) is now up on Github as a public repo, and feel free to email us at [info@rittmananalytics.com](mailto:info@rittmananalytics.com) for more information … or if you’d like us to bring this framework and our data centralization services to your organization. --- # Happy 10th Birthday Google BigQuery …! Canonical: https://rittmananalytics.com/blog/2020/05/20/happy-10th-birthday-google-bigquery URL: https://rittmananalytics.com/blog/2020/05/20/happy-10th-birthday-google-bigquery Description: \= s.session\_start\_tstampORDER BY w.id, s.session\_start\_tstamp) To return a user-specific measure of value that we can then use in our attribution model, we’ll create a simple query that returned each user’s ID and the total number of transactions they’d completed on the platform. SELECT user\_id AS user\_id,SUM(transactions) AS total\_user\_transactionsFROM \`ra-development.analytics.user\_transactions\`WHERE is\_approvedGROUP BY 1 Now comes the part when we start adding attribution logic into the SQL query using the following rules: - For each of the sessions for a given user, use the FIRST\_VALUE analytic SQL function to work out if that session is the first one in sequence for the user; if it is and we’re using the first click attribution method then allocate 100% of the conversion value to that session and none to the ones that follow, an attribution method that places all the value on the acquisition channel that first brought that user to our website - Similarly, use the LAST\_VALUE function on each user session row and allocate 100% of the conversion value to that row if we’re using the last click attribution method, and none for the ones that preceded it, placing all the value on the channel that eventually converted the visitor into a paying user - If we’re using the even click attribution method, count how many sessions the user took in-total to eventually convert and allocate the conversion value equally to each of them - And if we’re using the time decay attribution method, work out how many days did the visitor take to convert and then allocate 100% of the value to sessions in the 7 days up to conversion, 50% of the conversion value to those session between 7 and 14 days and halve and halve the value attributed again for sessions up to 21 and 28 days before conversion. Added to our SQL query as a CTE with the user transactions SELECT as a second CTE, the query to implement these attribution rules looks like this: WITH session\_attribution AS(SELECT \*,CASE WHEN session\_id = LAST\_VALUE(session\_id)OVER (partition by user\_id order by session\_start\_tstamp ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING) THEN 1ELSE 0END AS LAST\_click\_attrib\_pct,CASE WHEN session\_id = FIRST\_VALUE(session\_id)OVER (partition by user\_id order by session\_start\_tstamp ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING) THEN 1ELSE 0END AS first\_click\_attrib\_pct,1/count(session\_id) OVER (partition by worker\_id) AS even\_click\_attrib\_pct,CASE WHEN session\_start\_tstamp <\= timestamp\_sub(created\_at, interval 7 day) THEN 1WHEN session\_start\_tstamp \> timestamp\_sub(created\_at, interval 14 day) THEN .5WHEN session\_start\_tstamp \> timestamp\_sub(created\_at, interval 14 day) AND session\_start\_tstamp \> timestamp\_sub(created\_at, interval 21 day) THEN .25WHEN session\_start\_tstamp \> timestamp\_sub(created\_at, interval 21 day) AND session\_start\_tstamp \> timestamp\_sub(created\_at, interval 28 day) THEN .125ELSE 0END AS time\_decay\_attrib\_pctFROM(– sessions\_and\_registration\_dates\_CTE),user\_transactions AS(– user\_transactions\_CTE) Then we complete the attribution model query with a final SELECT from both CTEs at the end, multiplying the total transactions for each user by the various model percentages for each user session row coming from Segment, like this: SELECT session\_attribution.\*,total\_user\_transactions \* first\_click\_attrib\_pct AS first\_click\_attrib\_total\_transactions,total\_user\_transactions \* last\_click\_attrib\_pct AS last\_click\_attrib\_total\_transactions,total\_user\_transactions \* even\_click\_attrib\_pct AS even\_click\_attrib\_total\_transactions,total\_user\_transactions \* time\_decay\_attrib\_pct AS time\_decay\_attrib\_total\_transactions,1 \* first\_click\_attrib\_pct AS first\_click\_attrib\_registrations,1 \* last\_click\_attrib\_pct AS last\_click\_attrib\_registrations,1 \* even\_click\_attrib\_pct AS even\_click\_attrib\_registrations,1 \* time\_decay\_attrib\_pct AS time\_decay\_attrib\_registrationsFROM session\_attributionJOIN user\_transactionsON session\_attribution.user\_id = user\_transactions.user\_id The final, compete SQL query used for the attribution model is also available for download as a [Github gist here](https://gist.github.com/markrittman/2e7f0d530338e167f8255b5c083dbc76). Finally, we can add this query into dbt as a model or just plug it directly into Looker as a SQL derived table, then create reports such as this one that aggregate each week’s registrations by week and channel to show where each week’s new users came from, using just one of the attribution models: ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872562152-o7godb.png%3Fresize%3D1305%252C768%26ssl%3D1) Screenshot 2020-02-08 at 23.04.41.png Or you can create a Looker dashboard with attribution models selectable via a dashboard filter, allowing users to show the value of each channel, campaign and source over time and by share of revenue generated using either of the first click, last click, even click or time-decay attribution methods. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fimported-images%2F1752872562251-s01bzd.png%3Fresize%3D1739%252C979%26ssl%3D1) Screenshot 2020-02-08 at 23.11.55.png As well as building multi-touch attribution models using Google BigQuery and dbt, Rittman Analytics partners with Qubit, Segment, Stitch, Fivetran and Snowflake to offer a full range of data analytics and data centralization services for high-growth, data-rich businesses; see our [Data Centralizatio](https://rittmananalytics.com/data-centralisation)n and [Getting Started with dbt](https://rittmananalytics.com/getting-started-with-dbt) solutions pages for details or [contact us now](https://share.hsforms.com/1NdJIcJ61T2C_Sh8LofypGA2md7u) to find out more. [dbt](https://rittmananalytics.com/blog/tag/dbt)[Looker](https://rittmananalytics.com/blog/tag/Looker)[Stitch](https://rittmananalytics.com/blog/tag/Stitch) [Mark Rittman](https://rittmananalytics.com/blog?author=63c478ac6811232b3153d585) --- # Forecasting HubSpot Deal Revenue and Project Resourcing using dbt, Google BigQuery and Looker Canonical: https://rittmananalytics.com/blog/2020/01/07/forecasting-hubspot-deal-revenue-and-project-resourcing-using-dbt-google-bigquery-and-looker URL: https://rittmananalytics.com/blog/2020/01/07/forecasting-hubspot-deal-revenue-and-project-resourcing-using-dbt-google-bigquery-and-looker Description: Last week in our blog on using dbt and Looker to analyze Hubspot CRM data we talked about metrics that let us look back at the history of a deal… Author: Mark Rittman Published: 2020-01-07T01:38:09+01:00 Modified: 2020-01-07T01:38:09+01:00 --- Last week in our blog on using dbt and Looker to analyze Hubspot CRM data we talked about [metrics that let us look back at the history of a deal](https://rittmananalytics.com/blog/2020/1/2/modelling-slowly-changing-dimensions-type-23-and-6-using-dbt-and-looker) to understand deal velocity and measure open and active deals at any particular point in the past. In this blog I wanted to share how we use that same data to forecast our likely revenue in the coming months, and how we use this forecasting technique along with some custom Hubspot deal properties to predict what resources and skills we’ll need to deliver the projects we’re forecasting. In out Hubspot deals table, as processed previously in dbt tidy-up and cleanse the raw data coming from Hubspot via Stitch, a typical row of deal data contains these columns relevant to a revenue forecast: - The predicted start and end date of the deal, typically a consulting engagement of two or three two-week duration development sprints, along with the duration of the engagement in days - The sales funnel stage for the deal as of now; we’ve excluded those deals either lost, or won and already delivered - The predicted deal value, and a probability percentage based on current deal stage that when used as a modifier (multiplier) with the deal amount gives us a weighted pipeline deal value What we’d like to do is forecast, based on the estimated start and end dates of our open deals and the current weighted pipeline value of that deal, what our revenue numbers are likely to look like over the total duration of those deals. In [dbt](https://rittmananalytics.com/getting-started-with-dbt) we can do that by taking this deals data and using it to work out what revenue each deal brings in over the time it’s active, then joining those daily revenue amounts to a date array table we create on the fly to first aggregate each deal’s total revenue over those days, and then calculate the total revenue per month from all of those forecasted deals. {{config(materialized=’table’)}}WITH daily\_weighted\_revenue as (SELECT\*,(amount \* probability) / nullif(contract\_days,0) AS contract\_daily\_weighted\_revenue,amount / contract\_days AS contract\_daily\_full\_revenue,(amount / contract\_days) – ((amount \* probability) / contract\_days) AS contract\_diff\_daily\_revenueFROM (SELECT\*,TIMESTAMP\_DIFF(end\_date\_ts,start\_date\_ts,DAY) AS contract\_daysFROM (SELECTdealname,deal\_id,amount,probability,start\_date\_ts,end\_date\_tsFROM{ }WHEREstage\_label not in (‘Closed Lost’,’Closed Won and Delivered’)GROUP BY1,2,3,4,5,6,7))),months as (SELECT \*FROM UNNEST(GENERATE\_DATE\_ARRAY(‘2019-01-10’, ‘2024-01-01’, INTERVAL 1 DAY)) day\_ts)SELECT deal\_id,date\_trunc(day\_ts,MONTH) as month\_ts,sum(revenue\_days) as revenue\_days,sum(daily\_weighted\_revenue) as weighted\_amount\_monthly\_forecast,sum(daily\_full\_revenue) as full\_amount\_monthly\_forecast,sum(daily\_diff\_revenue) as diff\_amount\_monthly\_forecastfrom (SELECT deal\_id,day\_ts,count(\*) as revenue\_days,sum(contract\_daily\_weighted\_revenue) daily\_weighted\_revenue,sum(contract\_daily\_full\_revenue) daily\_full\_revenue,sum(contract\_diff\_daily\_revenue) daily\_diff\_revenueFROM months mJOIN daily\_weighted\_revenue dON TIMESTAMP(m.day\_ts) between d.start\_date\_ts and timestamp\_sub(d.end\_date\_ts, interval 1 day)GROUP BY 1,2)GROUP BY 1,2 Bringing the table that dbt creates for this SQL query into Looker and joining it to our main Hubspot-sourced deals table, we can now predict revenue over the coming few months based on start date and length of deal with each deal value weighted based on its current deal stage to allow for the fact that not all deals will close. In addition to recording revenue for our deals in Hubspot we also use custom deal properties to record aspects of the deal such as what products are likely to be part of the solution, how many sprints and of what type will be needed and who within the team we’re thinking of assigning the project to. Now I can predict overall and individual workload for each of our consulting team, like this: I can see how these forecasted projects are being priced: And finally, useful for training and recruitment planning, I can see what technology skills our team need to have to service these deals and when. For more details of what we can do for you with dbt, check the [new dbt Solutions page](https://rittmananalytics.com/getting-started-with-dbt) and [Data Centralization solution](https://rittmananalytics.com/data-centralisation) page on our website. --- # Modelling SCD Type 6 (1+2+3) Slowly Changing Dimensions using dbt and Looker Canonical: https://rittmananalytics.com/blog/2020/01/03/2020-1-3-modelling-slowly-changing-dimensions-type-23-and-6-using-dbt-and-looker URL: https://rittmananalytics.com/blog/2020/01/03/2020-1-3-modelling-slowly-changing-dimensions-type-23-and-6-using-dbt-and-looker Description: Like many of our clients we use Hubspot to track our sales opportunities, and use Stitch and dbt to extract and then transform data on these deals for analysis in… Author: Mark Rittman Published: 2020-01-03T01:00:33+00:00 Modified: 2020-01-03T01:00:33+00:00 --- Like many of our clients we use Hubspot to track our sales opportunities, and use [Stitch and dbt to extract and then transform data](https://rittmananalytics.com/blog/2019/5/7/how-rittman-analytics) on these deals [for analysis in Looker.](https://rittmananalytics.com/blog/2019/8/19/hubspot-data-actions-harvest-analytical-workflows-and-looker-data-platform) Stitch extracts data from Hubspot every 30 minutes and our Looker dashboards show the current state of our pipeline based on the each deal’s current funnel stagel; similarly our Looker explore allows users to slice-and-dice this pipeline as-of now by channel, technology type, resource profile and deal source. Together these give us useful insights into where our new business comes from today, what skills our team will need for deals just about to close and whether we’re hitting targets today for enquiries, open deals and closed business. But there’s also a category of sales metrics that require us to look back at the history of each deal over time to see whether we’re moving deals through the sales process fast enough, or to measure our sales metrics and customer counts today against those same metrics at earlier points in the year. Hubspot as you’d expect tracks the various stages that each deal goes through, as shown in the screenshot below of one of our deals that went from enquiry through to proposal, then PoC and finally to closed won business. In data warehouse data modelling terminology this is what’s called a [“slowly changing dimension”](https://en.wikipedia.org/wiki/Slowly_changing_dimension) (SCD), with our Looker model currently tracking just the current state of the deals dimension (SCD Type 1) and our need being to track the history of changes to that dimension (making it then a SCD Type 2 dimension). Creating a history of these changes requires us to somehow track when a change happens to the details of a deal, for example changing its status from open enquiry to closed business, and then write these changes to a “history” table for this dimension with valid-from and valid-to dates that allow us to view the status of the deal at any particular point in-time. [Our preferred data modelling and transformation tool](https://rittmananalytics.com/getting-started-with-dbt) is [dbt](https://www.getdbt.com/) (“Data Build Tool”) and it has a built-in feature for SCD2-style versioning of changes to a table called [Snapshots](https://blog.getdbt.com/track-data-changes-with-dbt-snapshots/). This feature takes a SQL SELECT statement and the names of columns we want to use as the unique key and updated\_at columns and then uses these to create a changed data capture process, automatically building for us a snapshot-versioned history of our deals table that looks like this: It’s then relatively straightforward to add this snapshot table back into Looker as another deals view that joins to the rest of our model on the deal\_id, for example, but then you’ve got two distinct LookML views within Looker that contain details on our sales deals, one of which points to a table that holds the current state of each deal and the other that points to our history table, making it tricky to answer questions such as “for each deal and it’s current deal state, what’s the average number of days its taken that deal to progress through its stages”. A neater solution would be to create what’s termed a “Type 6” slowly changing dimension, with the “6” based on the fact that it combines features of type 1 (track current state), type 2 (track historic state) and type 3 (track previous state) slowly changing dimensions in a single table (i.e. 1+2+3 = 6). Referencing the [Kimball University website article](https://www.kimballgroup.com/2013/02/design-tip-152-slowly-changing-dimension-types-0-4-5-6-7/) on Type 6 slowly changing dimensions: *“Type 6 builds on the type 2 technique by also embedding current attributes in the dimension so that fact rows can be filtered or grouped by either the type 2 value in effect when the measurement occurred or the attribute’s current value. The type 6 moniker was suggested by an HP engineer in 2000 because it’s a type 2 row with a type 3 column that’s overwritten as a type 1; both 2 + 3 + 1 and 2 x 3 x 1 equal 6. With this approach, the current attributes are updated on all prior type 2 rows associated with a particular durable key, as illustrated by the following sample rows:”* In dbt terms this means setting up our snapshot in the same way as before, using the batched\_at Stitch extract metadata field to select the most recent row from the Hubspot deal extract table, like this: /\*This snapshot table will live in:analytics.snapshots.deals\_snapshot\*/{% snapshot deals\_snapshot %}{{config(target\_schema=’snapshots’,strategy=’check’,unique\_key=’deal\_id’,check\_cols=’all’)}}with deals as (select \* from (select \*,MAX(\_sdc\_batched\_at) OVER(PARTITION BY dealidORDER BY \_sdc\_batched\_atRANGE BETWEEN UNBOUNDED PRECEDINGAND UNBOUNDED FOLLOWING) latest\_sdc\_batched\_atfrom{{ source(‘hubspot’, ‘deals’) }})where latest\_sdc\_batched\_at = \_sdc\_batched\_at),new\_deal as (selectproperties.dealname.value AS dealname,properties.hubspot\_owner\_id.value AS hubspot\_owner\_id,properties.dealstage.value AS dealstage,properties.dealstage.value as dealstage\_id,properties.dealstage.timestamp as dealstage\_ts,properties.pipeline.value AS pipeline,properties.closedate.value AS closedate,properties.createdate.value AS createdate,…dealid AS deal\_id,\_sdc\_batched\_atfrom deals)select \* from new\_deal{% endsnapshot %} Then we create a dbt model that reads from this snapshot and uses the snapshot row contents as-is for our SCD2 column values, then uses the LAG() OVER() [analytic function](https://cloud.google.com/bigquery/docs/reference/standard-sql/analytic-function-concepts) to get our SCD3 previous value columns and the LAST\_VALUE() OVER () analytic function to get our current value columns, like this: scd\_deals as (select– SCD2 “historical values” attributes\*,– SCD3 “previous value” attributeslag(dealstage\_ts) over(partition by deal\_idorder by dbt\_updated\_at) as previous\_dealstage\_ts,lag(stage\_displayorder) over(partition by deal\_idorder by dbt\_updated\_at) previous\_stage\_displayorder,lag(stage\_label) over(partition by deal\_idorder by dbt\_updated\_at) previous\_stage\_label,– SCD1 “current” value attributeslast\_value(dealstage\_ts) over(partition by deal\_idorder by dbt\_updated\_atROWS BETWEEN UNBOUNDED PRECEDINGAND UNBOUNDED FOLLOWING) as current\_dealstage\_ts,last\_value(dealname) over (partition by deal\_idorder by dbt\_updated\_atROWS BETWEEN UNBOUNDED PRECEDINGAND UNBOUNDED FOLLOWING) as current\_dealname,last\_value(dealstage\_id) over (partition by deal\_idorder by dbt\_updated\_atROWS BETWEEN UNBOUNDED PRECEDINGAND UNBOUNDED FOLLOWING) as current\_dealstage\_id,last\_value(stage\_label) over (partition by deal\_idorder by dbt\_updated\_atROWS BETWEEN UNBOUNDED PRECEDINGAND UNBOUNDED FOLLOWING) as current\_stage\_label,– and “original” value attributesfirst\_value(amount) over (partition by deal\_idorder by dbt\_updated\_atROWS BETWEEN UNBOUNDED PRECEDINGAND UNBOUNDED FOLLOWING) as original\_amountfrom joining),date\_difference as (select \*,timestamp\_diff(dealstage\_ts,previous\_dealstage\_ts, day)as days\_between\_stage,timestamp\_diff(current\_timestamp(),original\_createddate,day)as days\_since\_deal\_created,current\_amount – original\_amountas amount\_diff\_since\_deal\_createdfrom scd\_deals)select \* except (pipelineid,\_sdc\_batched\_at) from date\_difference Right at the end then you’ll see a final CTE that uses the current and previous deal stage timestamps to calculate days between funnel stages for each deal and stage in that deal’s progress through the funnel, and a final metric that measures days since the deal was first recorded. Now we can map just this single SCD6 table into our Looker model and have just the one view in our Looker explore for deals information, like this: And then using this new SCD6 table and Looker view we can create queries that bring together data on a deal now and across all time, like this: or we can compare the stage our deal is at now with its previous deal stage, like this: If one of our users just selects the regular dimension fields for a deal, the query uses our “current value” columns along with GROUP BY and SUM\_DISTINCT to return a single row for each deal and the correct measure numbers, without the user having to specifically ask for the current version of each deal. But if they want to see the deal history, they just select the historic value columns alongside the regular deal details to see them listed out versioned courtesy of our dbt snapshot table. For more details of what we can do for you with dbt, check the [new dbt Solutions page](https://rittmananalytics.com/getting-started-with-dbt) on our website. --- # Previewing the New Looker Dashboard Experience … and What’s Coming with Looker 7 Canonical: https://rittmananalytics.com/blog/2019/12/18/previewing-the-new-looker-dashboard-experience-and-whats-coming-with-looker-7 URL: https://rittmananalytics.com/blog/2019/12/18/previewing-the-new-looker-dashboard-experience-and-whats-coming-with-looker-7 Description: ?data= { "writeKey": "Uuel9igVwbCyM7y7TlT1sIQBs7Sli6hQ", "userId": "025pikachu025", "event": "Email Opened json", "properties": { "subject": "The Electric Daily", "email": "peekAtMe@email.poke" } } https://api.segment.io/v1/pixel/track?writeKey=Uuel9igVwbCyM7y7TlT1sIQBs7Sli6hQ&userId=025pikachu025&event=EmailOpened&properties.subject=The Electric Daily&traits.name=Brantley Beaird Then you include a reference to this img URL in the email you send out to subscribers in your campaign, and when they then open your email it registers in our case a tracking event in Segment Connections that’s then recorded in the user’s profile in Personas. ![](https://cdn-images-1.medium.com/max/800/0*zarOu13blgNApvuI) Capturing clicks on links in our campaign email is something we’ll do using the regular Segment Javascript tracker by passing details of the event(s) we want to capture in the link URL parameters, using the Javascript tracker’s [querystring API](https://segment.com/docs/connections/sources/catalog/libraries/website/javascript/#querystring-api). Note that you can pass both an identify event and a track event in the same querystring, useful if an email click is the first interaction your user has with Segment and you also want to log the email click as an event against that new user. ![](https://cdn-images-1.medium.com/max/800/0*MmLcnx2kL_Yljk5u) Finally, to complete the feedback loop we use Zapier and Segment’s HTTP API to capture all incoming emails to our info@rittmananalytics.com and other key email destinations and use that service to lookup the Hubspot contact IDs for the sender or recipient email address depending on who we want to track, and then send that this information to Segment Connections as a track event using Segment’s [HTTP API](https://segment.com/docs/connections/sources/catalog/libraries/server/http-api/). ![](https://cdn-images-1.medium.com/max/800/0*VKHOCGXsImB9w3M5) We can then see all of these events alongside other interactions each mailing list subscriber has made across all our other digital channels in their Personas event journey and user profile. ![](https://cdn-images-1.medium.com/max/800/0*JACETjFS3MC2ov2o) You can read more about our partnership with Segment [here](https://rittmananalytics.com/blog/2019/6/12/rittman-analytics-is-now-a-segment-certified-implementation-partner), or [schedule a call with us now](https://meetings.hubspot.com/mark-rittman) if you’re interested in building-out a customer data platform or using Segment Personas for this type of scenario. *Originally published at* [*https://rittmananalytics.com*](https://rittmananalytics.com/blog/2019/12/15/segmentcdpemailtracking) *on December 15, 2019.* --- # Using Segment Personas for Visitor Lead Scoring at Rittman Analytics Canonical: https://rittmananalytics.com/blog/2019/11/06/using-segment-personas-for-visitor-lead-scoring-at-rittman-analytics URL: https://rittmananalytics.com/blog/2019/11/06/using-segment-personas-for-visitor-lead-scoring-at-rittman-analytics Description: Rittman Analytics are a Certified Implementation Partner for Segment and we’ve implemented two of their services, Segment Connections and Segment Personas, on our two Squarespace-hosted websites; our main company site… Author: Mark Rittman Published: 2019-11-06T02:50:49+01:00 Modified: 2019-11-06T02:50:49+01:00 --- [Rittman Analytics are a Certified Implementation Partner for Segment](https://rittmananalytics.com/blog/2019/6/12/rittman-analytics-is-now-a-segment-certified-implementation-partner) and we’ve implemented two of their services, Segment Connections and Segment Personas, on our two Squarespace-hosted websites; our main company site at [https://rittmananalytics.com](https://rittmananalytics.com/) and the site we host the Drill to Detail Podcast website at, the subdomain [https://drilltodetail.com. ](https://drilltodetail.com. /); We use a single Segment Javascript tracker script installed on both sites that replaces all of the individual trackers used by Google Tag Manager, Hubspot, Google Analytics and other site analytics services we use, with the Segment tracker then routing all of our visitor interactions to these downstream platforms along with supporting customer data from Hubspot and mailing list sends, opens and email link clicks from Mailchimp. We’ve configured Segment to use the same same Segment property ID across both sites so that visitors keep the same unique visitor ID across both [https://rittmananalytics.com](https://rittmananalytics.com/) and [https://drilltodetail.rittmananalytics.com](https://drilltodetail.rittmananalytics.com,/) so that when page view and tracking data from the two sites makes its way into Google BigQuery for analysis later on, we can measure the extent to which promotions for our company on the podcast website drive traffic to our main company site, using SQL like this: SELECTreceived\_at,context\_page\_url,context\_page\_referrerFROM\`ra-development.company\_website.pages\`WHEREcontext\_page\_referrer LIKE ‘%drilltodetail%’AND context\_page\_url LIKE ‘%https://rittmananalytics.%’ORDER BY1 We also auto-track all page views, link clicks and button click events in a similar way to Heap and Mixpanel’s autotrack features, using code such as the example below injected into the footer section of all of our site page. < script \> var links \= document . getElementsByTagName ( ‘a’ ); for ( i \= 0 ; i < links . length ; i ++ ) { links \[ i \]. addEventListener ( ‘click’ , function () { analytics . track ( ‘Clicked a Link’ , { target : $ ( this ). attr ( “href” ), title : document . title , label : $ ( this ). attr ( “innerText” ), Link\_text : $ ( this ). attr ( “text” ), baseURI : $ ( this ). attr ( “baseURI” )});});}; As well as using [Segment Connections](https://segment.com/product/connections/) to route customer and visitor data from our website and other sources into our downstream marketing applications, we’ve also started to use [Segment Personas](https://segment.com/product/personas/) to build-up a 360-degree view of each visitor’s interactions on our site and score those visitors by their potential to become a sales lead. We do this through two features that Personas provides; [computed traits](https://segment.com/docs/personas/computed-traits/) and [SQL traits](https://segment.com/docs/personas/sql-traits/). Traits are properties that we assign or compute for visitors to our site and computed traits allows us, for example, to calculate a running total of how many non-podcast and non-homepage pages that a visitor has looked at on our websites. Computed traits can be counts of certain event types filtered on a property value; aggregations (sums for example) of revenue values in transaction events, unique lists of event property values such as unique podcast episodes and can be calculated over a sliding window of days, in our case 30 days. Individual computed traits can then be combined in more complex calculations together with external data in SQL traits, such as the one we use to calculate our engaged prospect score. Finally, we can define an audience in Personas based on this engaged prospect visitor score that includes all visitors with a score above a certain value based on their last 30 days activity. This audience is then made available for sending to downstream destinations along with other audiences based on individual computed trait conditions. You can read more about our partnership with Segment [here](https://rittmananalytics.com/blog/2019/6/12/rittman-analytics-is-now-a-segment-certified-implementation-partner), or [schedule a call with us now](https://meetings.hubspot.com/mark-rittman) if you’re interested in building-out a customer data platform or using Segment Personas for this type of scenario. --- # Using Segment Personas for Visitor Lead Scoring at Rittman Analytics Canonical: https://rittmananalytics.com/blog/2019/11/06/2019-11-6-e058gepqwiyyx4mop3eb06mn9ckjkp URL: https://rittmananalytics.com/blog/2019/11/06/2019-11-6-e058gepqwiyyx4mop3eb06mn9ckjkp Description: Rittman Analytics are a Certified Implementation Partner for Segment and we’ve implemented two of their services, Segment Connections and Segment Personas, on our two Squarespace-hosted websites; our main company site… Author: Mark Rittman Published: 2019-11-06T01:50:49+00:00 Modified: 2019-11-06T01:50:49+00:00 --- [Rittman Analytics are a Certified Implementation Partner for Segment](https://rittmananalytics.com/blog/2019/6/12/rittman-analytics-is-now-a-segment-certified-implementation-partner) and we’ve implemented two of their services, Segment Connections and Segment Personas, on our two Squarespace-hosted websites; our main company site at [https://rittmananalytics.com](https://rittmananalytics.com/) and the site we host the Drill to Detail Podcast website at, the subdomain [https://drilltodetail.com. ](https://drilltodetail.com. /); We use a single Segment Javascript tracker script installed on both sites that replaces all of the individual trackers used by Google Tag Manager, Hubspot, Google Analytics and other site analytics services we use, with the Segment tracker then routing all of our visitor interactions to these downstream platforms along with supporting customer data from Hubspot and mailing list sends, opens and email link clicks from Mailchimp. We’ve configured Segment to use the same same Segment property ID across both sites so that visitors keep the same unique visitor ID across both [https://rittmananalytics.com](https://rittmananalytics.com/) and [https://drilltodetail.rittmananalytics.com](https://drilltodetail.rittmananalytics.com,/) so that when page view and tracking data from the two sites makes its way into Google BigQuery for analysis later on, we can measure the extent to which promotions for our company on the podcast website drive traffic to our main company site, using SQL like this: SELECTreceived\_at,context\_page\_url,context\_page\_referrerFROM\`ra-development.company\_website.pages\`WHEREcontext\_page\_referrer LIKE ‘%drilltodetail%’AND context\_page\_url LIKE ‘%https://rittmananalytics.%’ORDER BY1 We also auto-track all page views, link clicks and button click events in a similar way to Heap and Mixpanel’s autotrack features, using code such as the example below injected into the footer section of all of our site page. < script \> var links \= document . getElementsByTagName ( ‘a’ ); for ( i \= 0 ; i < links . length ; i ++ ) { links \[ i \]. addEventListener ( ‘click’ , function () { analytics . track ( ‘Clicked a Link’ , { target : $ ( this ). attr ( “href” ), title : document . title , label : $ ( this ). attr ( “innerText” ), Link\_text : $ ( this ). attr ( “text” ), baseURI : $ ( this ). attr ( “baseURI” )});});}; As well as using [Segment Connections](https://segment.com/product/connections/) to route customer and visitor data from our website and other sources into our downstream marketing applications, we’ve also started to use [Segment Personas](https://segment.com/product/personas/) to build-up a 360-degree view of each visitor’s interactions on our site and score those visitors by their potential to become a sales lead. We do this through two features that Personas provides; [computed traits](https://segment.com/docs/personas/computed-traits/) and [SQL traits](https://segment.com/docs/personas/sql-traits/). Traits are properties that we assign or compute for visitors to our site and computed traits allows us, for example, to calculate a running total of how many non-podcast and non-homepage pages that a visitor has looked at on our websites. Computed traits can be counts of certain event types filtered on a property value; aggregations (sums for example) of revenue values in transaction events, unique lists of event property values such as unique podcast episodes and can be calculated over a sliding window of days, in our case 30 days. Individual computed traits can then be combined in more complex calculations together with external data in SQL traits, such as the one we use to calculate our engaged prospect score. Finally, we can define an audience in Personas based on this engaged prospect visitor score that includes all visitors with a score above a certain value based on their last 30 days activity. This audience is then made available for sending to downstream destinations along with other audiences based on individual computed trait conditions. You can read more about our partnership with Segment [here](https://rittmananalytics.com/blog/2019/6/12/rittman-analytics-is-now-a-segment-certified-implementation-partner), or [schedule a call with us now](https://meetings.hubspot.com/mark-rittman) if you’re interested in building-out a customer data platform or using Segment Personas for this type of scenario. --- # Presenting on Looker Data Platform and Oracle Autonomous Data Warehouse at the UKOUG Analytics Modernisation Summit – tomorrow, 8th Oct. 2019 Canonical: https://rittmananalytics.com/blog/2019/10/07/presenting-on-looker-data-platform-and-oracle-autonomous-data-warehouse-at-the-ukoug-analytics-modernisation-summit-tomorrow-8th-oct-2019 URL: https://rittmananalytics.com/blog/2019/10/07/presenting-on-looker-data-platform-and-oracle-autonomous-data-warehouse-at-the-ukoug-analytics-modernisation-summit-tomorrow-8th-oct-2019 Description: Oracle Autonomous Data Warehouse Cloud (ADWC) brings elastic-scaling, easy cloud provisioning and Exadata performance improvements to Oracle Data Warehousing customers, making this new Oracle Database Cloud technology an ideal database… Author: Mark Rittman Published: 2019-10-07T23:14:56+01:00 Modified: 2019-10-07T23:14:56+01:00 --- Oracle Autonomous Data Warehouse Cloud (ADWC) brings elastic-scaling, easy cloud provisioning and Exadata performance improvements to Oracle Data Warehousing customers, making this new Oracle Database Cloud technology an ideal database storage and compute layer to work with the next-generation Looker Data Platform. Our use of Oracle ADWC on client projects was [profiled in an article in Forbes](https://www.forbes.com/sites/oracle/2018/10/04/how-a-big-business-can-use-autonomous-it-to-move-like-a-startup/#2e8b685c1752) last year and we’ve spoken at events such as Oracle Open World and the UK Oracle User Group on [the agility that Oracle ADWC brings to enterprises](https://rittmananalytics.com/blog/2018/11/14/slides-and-forbescom-article-from-data-warehouse-like-a-tech-startup-with-oracle-autonomous-data-warehouse-cloud) that had previously only been possible at small tech startups. “How A Big Business Can Use An Autonomous Database To Move Like A Startup” — [Forbes.com](https://www.forbes.com/sites/oracle/2018/10/04/how-a-big-business-can-use-autonomous-it-to-move-like-a-startup/#13d4ce0f1752) But it’s no use bringing agility to your data warehousing platform if your analytics are still stuck in the old, monolithic era when projects take months or years to complete rather than just a few weeks and your developers are forced to work together by sharing binary repository files in Dropbox with version numbers appended to the file name … [a process I know all too well](https://www.amazon.co.uk/Oracle-Business-Intelligence-Developers-Guide-ebook/dp/B009E71X0G) having written the book on it back in 2012. [Looker](https://rittmananalytics.com/home-index#looker) brings a truly modern and agile software and content development approach to business analytics taking concepts such as the business semantic model, single version of the truth and metadata-driven native SQL generation and updating it for today’s multi-cloud, speed-of-thought analytics tool users. If you’re an Oracle Database or Applications user and you’re interested in how Looker Data Platform can make your analytics truly agile and enable your multi-cloud strategy, I’m speaking on this topic at the [UK Oracle User Group Analytics Modernisation Summit](https://ukoug.org/page/ukougreporting) in London tomorrow, Tuesday, 8th October 2019 at 2.10pm. In [“Democratise your Oracle Data Analytics using Oracle ADWC and Looker Data Platform”](https://ukoug.org/page/analyticsagenda) I’ll outline the key features of Looker in the context of traditional ways of developing BI content on-top of Oracle database source data and highlight key areas in which Looker leverages Oracle ADWC and Oracle 18c/19c database features. I’ll also show how Looker can work across multiple cloud providers without lock-in or extensive configuration, and explain how Looker development really does deliver on the promise of agile development and rapid time-to-value. Until then, you can find out more about our [data analytics, data engineering and data strategy services](https://rittmananalytics.com/home-index#looker) for [Oracle, Looker, Snowflake and Google Cloud Platform customers](https://rittmananalytics.com/customers) on our company website, or contact me at [mark.rittman@rittmananalytics.com](mailto:mark.rittman@rittmananalytics.com) to find out more now. --- # Presenting on Oracle Autonomous Data Warehouse Cloud and Looker at the UK Oracle User Group… Canonical: https://rittmananalytics.com/blog/2019/10/07/presenting-on-oracle-autonomous-data-warehouse-cloud-and-looker-at-the-uk-oracle-user-group URL: https://rittmananalytics.com/blog/2019/10/07/presenting-on-oracle-autonomous-data-warehouse-cloud-and-looker-at-the-uk-oracle-user-group Description: Oracle Autonomous Data Warehouse Cloud (ADWC) brings elastic-scaling, easy cloud provisioning and Exadata performance improvements to Oracle Data Warehousing customers, making this new Oracle Database Cloud technology an ideal database… Author: Mark Rittman Published: 2019-10-07T09:00:00+01:00 Modified: 2026-07-02T00:35:07+01:00 --- ![Presenting on Oracle Autonomous Data Warehouse Cloud and Looker at the UK Oracle User Group…](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F294b0-1hglnj9frktk6qj6ohtuh7g.png) Oracle Autonomous Data Warehouse Cloud (ADWC) brings elastic-scaling, easy cloud provisioning and Exadata performance improvements to Oracle Data Warehousing customers, making this new Oracle Database Cloud technology an ideal database storage and compute layer to work with the next-generation Looker Data Platform. Our use of Oracle ADWC on client projects was [profiled in an article in Forbes](https://www.forbes.com/sites/oracle/2018/10/04/how-a-big-business-can-use-autonomous-it-to-move-like-a-startup/#2e8b685c1752) last year and we’ve spoken at events such as Oracle Open World and the UK Oracle User Group on [the agility that Oracle ADWC brings to enterprises](https://rittmananalytics.com/blog/2018/11/14/slides-and-forbescom-article-from-data-warehouse-like-a-tech-startup-with-oracle-autonomous-data-warehouse-cloud) that had previously only been possible at small tech startups. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F294b0-1hglnj9frktk6qj6ohtuh7g.png) “How A Big Business Can Use An Autonomous Database To Move Like A Startup” — [Forbes.com](https://www.forbes.com/sites/oracle/2018/10/04/how-a-big-business-can-use-autonomous-it-to-move-like-a-startup/#13d4ce0f1752) But it’s no use bringing agility to your data warehousing platform if your analytics are still stuck in the old, monolithic era when projects take months or years to complete rather than just a few weeks and your developers are forced to work together by sharing binary repository files in Dropbox with version numbers appended to the file name … [a process I know all too well](https://www.amazon.co.uk/Oracle-Business-Intelligence-Developers-Guide-ebook/dp/B009E71X0G) having written the book on it back in 2012. [Looker](https://rittmananalytics.com/home-index#looker) brings a truly modern and agile software and content development approach to business analytics taking concepts such as the business semantic model, single version of the truth and metadata-driven native SQL generation and updating it for today’s multi-cloud, speed-of-thought analytics tool users. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2Febaf3-1osuwcrkinvszfpew9rspea.png) Familiar … and yet modern, modular and agile : Looker Data Platform If you’re an Oracle Database or Applications user and you’re interested in how Looker Data Platform can make your analytics truly agile and enable your multi-cloud strategy, I’m speaking on this topic at the [UK Oracle User Group Analytics Modernisation Summit](https://ukoug.org/page/ukougreporting) in London on Tuesday, 8th October 2019. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2Fe4c2c-1u6zbxjdn4-3msvkskv3iha.png) Presenting on big data and analytics modernisation at UKOUG Tech’17, Birmingham In [“Democratise your Oracle Data Analytics using Oracle ADWC and Looker Data Platform”](https://ukoug.org/page/analyticsagenda) I’ll outline the key features of Looker in the context of traditional ways of developing BI content on-top of Oracle database source data and highlight key areas in which Looker leverages Oracle ADWC and Oracle 18c/19c database features. ![](https://cdn-images-1.medium.com/max/800/0*F84xGYnjRPwOMZoG) I’ll also show how Looker can work across multiple cloud providers without lock-in or extensive configuration, and explain how Looker development really does deliver on the promise of agile development and rapid time-to-value. Until then, you can find out more about our [data analytics, data engineering and data strategy services](https://rittmananalytics.com/home-index#looker) for [Oracle, Looker, Snowflake and Google Cloud Platform customers](https://rittmananalytics.com/customers) on our company website, or contact me at [mark.rittman@rittmananalytics.com](mailto:mark.rittman@rittmananalytics.com) to find out more now. --- # Rittman Analytics and dbt : Consulting Partnership and News on dbt Learn London Canonical: https://rittmananalytics.com/blog/2019/09/16/rittman-analytics-and-dbt-consulting-partnership-and-news-on-dbt-learn-london URL: https://rittmananalytics.com/blog/2019/09/16/rittman-analytics-and-dbt-consulting-partnership-and-news-on-dbt-learn-london Description: We’re very pleased to announce that Rittman Analytics is now an official Consulting Partner for dbt, working with our clients, the community and the wider dbt ecosystem to get the… Author: Mark Rittman Published: 2019-09-16T09:00:00+01:00 Modified: 2026-07-02T00:36:07+01:00 --- ![Rittman Analytics and dbt : Consulting Partnership and News on dbt Learn London](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F06b22-1oxpbvftkra7vwv6zvg0dwq.png) We’re very pleased to announce that [Rittman Analytics is now an official Consulting Partner for dbt](https://www.getdbt.com/ecosystem/), working with our clients, the community and the wider dbt ecosystem to get the most out of this open-source analytics framework and accompanying commercial dbtCloud service. We also have news of a special [dbt Learn training event](https://www.eventbrite.com/e/dbt-learn-october-2019-tickets-70286882959) taking place in London on October 24th and 25th 2019 with Rittman Analytics sponsoring the event reception … and a special registration discount for readers of our blog. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F06b22-1oxpbvftkra7vwv6zvg0dwq.png) ### So What, Exactly, is dbt? [As Tristan Handy wrote in a blog post](https://blog.getdbt.com/what--exactly--is-dbt-/) around this time last year, > “[dbt](https://www.getdbt.com/) (data build tool) is an open-source tool and accompanying commercial web service that enables data analysts and engineers to transform data in their warehouses more effectively and own the entire analytics engineering workflow. > dbt is the “T” in the Extract, Load and Transform (“ELT”) data architecture. It doesn’t extract or load data, but it’s extremely good at transforming data that’s already loaded into your warehouse. This “transform after load” architecture is becoming known as ELT (extract, load, transform), and dbt fits nicely into the modern BI stack, coupling with products like Looker, Stitch, Fivetran, Google BigQuery, Github and Snowflake Data Warehouse.” ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F5fcb8-0q_gav-zsz7lf4ohe.png) Today, dbt has ~280 companies using it in production, including companies like Casper, Seatgeek, and Wistia in the US and several of our clients in UK and Europe — and ourselves, as we talk about in our recent blog post on [How Rittman Analytics does Analytics](https://rittmananalytics.com/blog/2019/5/7/how-rittman-analytics). To accompany the dbt open-source toolkit and available as a paid-for cloud service, dbtCloud is used for managing your analytics engineering workflow in one place and includes features for scheduling, monitoring and creating CI/CD (“Continuous Integration/Continuous Delivery”) test pipelines. ### dbt Learn London 24/25–10–2019 : Special Offer dbt Learn is a two-day, in-person training program [running in London on October 24th and 25th 2019](https://www.eventbrite.com/e/dbt-learn-october-2019-tickets-70286882959) that teaches analysts the tools and knowledge needed to own the entire analytics engineering workflow. Taught by the Fishtown Analytics team and with Rittman Analytics as event reception sponsors, analysts will walk away with a practical foundation in data warehouse management, modeling strategies, automated testing, performance optimisation, and much more. The only technical requirement is a practical knowledge of SQL. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2Fda2da-0pebnr0iss646wisd.png) You can read more about dbt Learn in [this blog post from Fishtown Analytics](https://blog.getdbt.com/dbt-learn-analytics-engineering-for-data-analysts/), and use the promo code RITTMANANALYTICS when you [register on the event website](https://www.eventbrite.com/e/dbt-learn-october-2019-tickets-70286882959) to get $200 off of the standard course price. Spaces are strictly limited, and you can join us for a drinks and canapés reception at the end of the first day to network and share your dbt experiences — more details are on the event page [here](https://www.eventbrite.com/e/dbt-learn-october-2019-tickets-70286882959). ### Rittman Analytics and dbt Rittman Analytics has implemented dbt and dbtCloud on several client projects in scenarios such as - transforming event-based (Segment, mParticle), data pipeline (Stitch, Fivetran) and batch-loaded raw data sourced from SaaS, Telco, web application and other data sources - using Snowflake Data Warehouse, Postgres and Google BigQuery as sources/targets - integrating with GitHub, AWS CodeCommit for local/remote branch-based git development - creation of automated CI/CD dbt build test pipelines using dbtCloud and AWS CodeBuild/CodePipeline Finally, if you’re new to dbt or interested in how we use dbt and dbtCloud both internally and on client projects, check out our three recent blog posts on this topic to get a sense of what dbt and our services can do for your business. - [How Rittman Analytics does Analytics: Modern BI Stack Operational Analytics using Looker, Stitch, dbt and Google BigQuery](https://rittmananalytics.com/blog/2019/5/7/how-rittman-analytics) - [Customer Journey and Lifetime Value Analytics using Looker, Google BigQuery, Stitch and dbt](https://rittmananalytics.com/blog/2019/5/22/0r1fgtifyovghse903ha3vwdwwbp7j) - [Continuous Integration and Automated Build Testing with dbtCloud](https://rittmananalytics.com/blog/2019/6/11/continuous-integration-feature-branches-and-automated-build-tests-using-dbtcloud) --- # HubSpot Data Actions, Harvest Analytical Workflows and Looker Data Platform Canonical: https://rittmananalytics.com/blog/2019/08/19/hubspot-data-actions-harvest-analytical-workflows-and-looker-data-platform URL: https://rittmananalytics.com/blog/2019/08/19/hubspot-data-actions-harvest-analytical-workflows-and-looker-data-platform Description: At Rittman Analytics we use a number of SaaS applications to run our back office operations including HubSpot CRM for sales tracking, Harvest for timesheets and resource planning, Jira for… Author: Mark Rittman Published: 2019-08-19T01:25:17+01:00 Modified: 2019-08-19T01:25:17+01:00 --- At Rittman Analytics we use a number of SaaS applications to run our back office operations including [HubSpot CRM](https://www.hubspot.com/products/crm) for sales tracking, [Harvest](https://www.getharvest.com/harvest-time-tracking?adgr=29583830489&camp=445034489&chan=Google&device=c&gclid=CjwKCAjw7uPqBRBlEiwAYDsr19SoD5J3238XvE_28bGWtJHEBkfoOLNOP_1LJTJqP8kQLF5i400rLBoCYxkQAvD_BwE&kw=harvestapp&loc=1006812&type=brand) for timesheets and resource planning, [Jira](https://www.atlassian.com/software/jira) for project management and [Xero](https://xero.com/) for invoicing and accounting. [Back in May 2019 I blogged about](https://rittmananalytics.com/blog/2019/5/7/how-rittman-analytics) how we used services from Stitch, Fivetran and dbt to extract, transform and combine data from those services into a data warehouse running on Google BigQuery and since then we’ve built-out a number of finance, operations and customer success dashboards that provide our back office team with valuable insights into the performance of our business. But Looker is more than just an ad-hoc query and dashboarding tool, [it’s a full data platform](https://looker.com/platform/overview) that integrates with SaaS applications such as Harvest, HubSpot, Xero and Salesforce at multiple levels: - At a data-source level, querying and analyzing data from those application sources - Within the Looker explore UI using features such as links, data actions and the Action Hub - Within business workflows, providing analytical insights and trusted metrics through the Looker API ## Adding Links to HubSpot Deal and Company Pages as Explore Menu Items One example of this is how we’ve added menu options in the Looker explore to provide direct links into HubSpot for viewing the details of a deal or a company within that application, and for our sales manager to request an update on a deal where there’s been no contact from either side for a while. This type of simple integration is done through the “[link](https://docs.looker.com/reference/field-params/link)” property in LookML that you can add to dimension and measure definitions, as I’ve done in the example below for our HubSpot opportunity name dimension. dimension: opportunity\_name {type: stringsql: $.dealname ;;link: {label: “View Deal in Hubspot”url: “[https://app.hubspot.com/contacts/4402794/deal/](https://app.hubspot.com/contacts/4402794/deal/){ }/”icon\_url: “[http://app.hubspot.com/favicon.ico”](http://app.hubspot.com/favicon.ico”);}} Then when a user clicks on a field in an explore or a dashboard tile they’ll be able to click-through to HubSpot to view all of the context of the deal; if like us you use Google Apps authentication for both Looker and our SaaS applications you’ll be taken straight into HubSpot without having to login first. ## Creating Looker Data Actions to Update HubSpot Deal Properties A more advanced form of integration between Looker and HubSpot that we’ve setup is the ability for users to update details of a HubSpot deal directly from within Looker, a good example of how you can make your Looker content more “actionable”. This type of integration involves another type of LookML dimension and measure property called an “[action](https://looker.com/platform/actions)”. I first talked about actions in a blog post titled [“Using Looker Data Actions to Make Monzo Spend Analysis More Interactive … and Actionable”](https://rittmananalytics.com/blog/2018/05/02/using-looker-data-actions-to-make-monzo-spend-analysis-more-interactive-and-actionable) back in 2018 where I used the feature to call services such as Google Maps, Google My Business and via Zapier, Google Tasks; in the case of our HubSpot integration we’ve again use Zapier for the [webhook](https://zapier.com/page/webhooks/) service that our data actions call, setting up a three stage Zapier “Zap” that takes the details of the deal and the values that need changing from the webhook payload, then searches for the deal in HubSpot that needs updating and then applies those changes to the deal record. ## Adding Data Action Menu Items to Request HubSpot Deal Updates via GMail Another example of this type of data action integration we’ve setup is between Looker, HubSpot and GMail, giving us the ability to send an email to request a status update from the team directly from the Looker dashboard. ## Leveraging the Looker API to Provide KPIs for Harvest Workflows Automated using Zapier As well as providing hooks into HubSpot for managing the deal process directly from within Looker, we also use Looker to inform and provide trusted business key performance indicators (KPIs) for our automated workflows. An example of this is when we’ve closed a new deal and the salesperson or operations team then sets up a new project in Harvest to record our time and expenses against, and something we want to keep a close eye on is situations where a salesperson has closed the deal but done so by agreeing a daily rate below the rate we’d normally charge. To keep an eye on this type of deal being agreed and to make sure the salesperson involved knows that the discount will be coming out of their commission, we’ve setup another Zapier workflow that detects new Harvest projects being created, runs a look within Looker that returns the average rate charged across all projects over the past three months and sends a message to the salesperson concerned if the deal rate is below this average. In the future we can change how this average rate is calculated, introduce other data and analytics into the calculation and by using Looker we’re making sure these analytic insights are governed and delivered “as-a-service” to the requesting business process. If you’re looking to use Looker Data Platform to integrate, inform and help orchestrate your business processes then [drop us an email](mailto:sales@rittmananalytics.com) and we’d be happy to share our experiences so far. --- # Xero Financial Reporting in Looker using G-Accon for Xero, BigQuery and dbt Canonical: https://rittmananalytics.com/blog/2019/08/04/xero-financial-reporting-in-looker-using-g-accon-for-xero-bigquery-and-dbt URL: https://rittmananalytics.com/blog/2019/08/04/xero-financial-reporting-in-looker-using-g-accon-for-xero-bigquery-and-dbt Description: We use Xero as our accounting package at Rittman Analytics and thought it’d be useful to bring in some of our key financial and performance metrics from Xero’s Profit and… Author: Mark Rittman Published: 2019-08-04T23:21:28+01:00 Modified: 2019-08-04T23:21:28+01:00 --- We use [Xero](https://www.xero.com/) as our accounting package at Rittman Analytics and thought it’d be useful to bring in some of our key financial and performance metrics from [Xero’s Profit and Loss report](https://central.xero.com/s/article/Profit-and-Loss-New#Newversion) (Income Statement for our US readers) into our Looker environment, so that we can monitor gross and net revenue trends over time, check expenses and staff costs are within budget and keep an eye on our most important financial KPI, Net Margin %. Although Xero is a supported data source for Stitch, Fivetran and most other data pipelines-as-a-service, trying to reconstruct a P&L report from all of the Xero API tables whilst applying manual journals, categorising accounts into revenue or costs and coming-up with the same numbers as Xero’s own reports is fairly tricky; another approach that we’ve used with success is to use a Google Sheets add-in such as [G-Accon for Xero](https://www.accon.services/gacconforxero.html) to schedule exports of Xero’s P&L report into a Google Sheet and then access the results using a Google BigQuery external table. When you’ve set the Xero report export up within Google Sheets and G-Accon for Xero, the next step is to setup a [federated table in Google BigQuery](https://cloud.google.com/bigquery/external-data-drive) that maps to the Google Sheets sheet that you’ve just setup. Remember to share the Google Sheets file with the GCP service account that your Looker connection uses when accessing Google BigQuery data otherwise you’ll get a permissions issue when trying to query the Sheets data from Looker. Our next step was then to use [dbt](https://www.getdbt.com/) (“Data Build Tool”) to transform Xero’s account-level P&L numbers as landed into our base BigQuery federated tables into aggregated and derived revenue, cost of sales and overheads KPIs along with net and gross profit, margins and so on. Now our financial numbers and P&L account details are in Looker as additional views within our main operational analytics LookML model we can start by reproducing the original Profit & Loss report using Looker’s new beta-release [Table-Next visualization](https://docs.looker.com/exploring-data/visualizing-query-results/table-next-options), giving us subtotals for the account categories as you’d normally expect to see in a P&L-style financial report (numbers obfuscated, in-case you were wondering). Then to accompany the detail-level P&L report we then created a financial performance dashboard, some of the contents coming from the derived and aggregated financial KPIs table we created beforehand in dbt, some of the numbers (discretionary expense tracking over time for example) coming from the base P&L report numbers. Note how we’ve used Looker’s recently-added [trellis charting feature](https://discourse.looker.com/t/how-do-you-use-the-trellis-charts/12059) to break expenses spending out by the discretionary spend categories we’re most interested in, I think this works well for this type of analysis. Whether you go down the path of full Xero data integration into your warehouse using a service such as Stitch or Fivetran or if you choose the report-based export approach we used here, really comes down to whether you need granular access to every bit of your accounting data or some high-level (and processed) numbers as we did in this particular example. We’ve done both on projects for customers and ourselves in the past, so if you’re interested in bringing in Xero or other financial accounting data into your Looker analytics platform, [send us an email](mailto:info@rittmananalytics.com) or call our office on +44 1273 234690 and we’ll be happy to talk through the various options with you. --- # New Features in Looker 6.16 : Conditional Alerts (Beta), Content Curation (Beta) and LookML IDE Folders Canonical: https://rittmananalytics.com/blog/2019/07/18/new-features-in-looker-6-16-conditional-alerts-beta-content-curation-beta-and-lookml-ide-folders URL: https://rittmananalytics.com/blog/2019/07/18/new-features-in-looker-6-16-conditional-alerts-beta-content-curation-beta-and-lookml-ide-folders Description: Looker release 6.16 is rolling-out to customers right now and comes with three new features that you might find interesting: Conditional Alerts (beta), Content Creation (beta), and Folders with the… Author: Mark Rittman Published: 2019-07-18T22:01:09+01:00 Modified: 2019-07-18T22:01:09+01:00 --- Looker release 6.16 is rolling-out to customers right now and comes with three new features that you might find interesting: - Conditional Alerts (beta), - Content Creation (beta), and - Folders with the LookML IDE Note that to enable beta features such as conditional alerts and content creation boards in Looker you’ll need to log in as an admin to specifically enable them before they’re available for use, and as experimental features they are not fully developed and may significantly change or be completely removed in future releases. ## Conditional Alerts (Beta) You can create a form of “alert” in Looker right now by scheduling a look to be sent to you only when a report returns rows, doesn’t return rows or returns a different set of rows than last time, but this is a pretty obscure feature that most end-users won’t have heard of. The new Conditional Alerts (beta) feature in Looker 6.16 makes creating alerts much simpler and is enabled from the **Labs** section of the **Admin** page in Looker. Once you’ve enabled this beta feature, you’ll then see “alert” icons in the top right-hand corner of every dashboard item, like this: Click on the bell icon to define the alert. In the example below I’ve configured the alert to send me an email if net margin has dropped below 20% at the end of the previous day, with the email going out at 9am next day to give me a heads-up that action needs to be taken – but also saving me the time of scanning the dashboard each day just to the KPI is within target. Alerts created by users are now listed alongside scheduled looks on the Looker Admin page together with a history of previous alert executions. ## Content Curation (Beta) Boards are a new type of content object within Looker 6.16 and give you the ability to “pin” dashboards and looks relating to a particular topic or department together in a single place. Like conditional alerts, the new Content Creation / Boards (Beta) feature has to be enabled by an admin before it becomes available to end-users, but once you’ve done so then a new **Boards (beta)** menu item appears in the **Browse** page menu. Click on the **Pin a Dashboard or Look** button to bring up a dialog listing all of the dashboards and looks in spaces you’re able to access, like this: You can then add more dashboards or looks, and create sections within each board to further organize your content, as I’ve done here with our internal analytics board. ## IDE Folders The final new feature I wanted to highlight isn’t a beta, but it does need to be enabled for each individual LookML project before use. To switch on the new folders feature for your LookML project first select **Project Settings** from the drop-down menu in the LookML developer IDE: Then, check the **Enable Folders** checkbox just above the **Save Project Settings** button. Now, press the “+” button to see a new **Create Folder** option. Then once you’ve given your folder a name, you can move existing view, model and other LookML object files into the folder you’ve just created. Note how the various LookML files have different icons now for views, models etc. [Release notes for Looker 6.16](https://discourse.looker.com/t/looker-6-16-release-notes/12994?mkt_tok=eyJpIjoiTVRNMU1EZ3laVFk1WkRVNCIsInQiOiJrSEJmd1N2MjhiUWJzMTgwdUs2cExGWmxoczVMek9xajNnTk9BTEgxXC9XOVhFcVJPYzBOZ3d4T2ZJTGNXQ1JpSTc2bit1TitVZGljQ0U2SUt3NGVSYWZ4MGVhRGl3T3NWbDRjazJtbTRQS1lTVHRWVG9qd1p5RFRkN1ljUUV1TzQifQ%3D%3D#heading--folders) are on the Looker website, and [contact us now](mailto:info@rittmananalytics.com) if you’d like to find out how we can help you get moving with your Looker implementation. --- # Rittman Analytics is now a UK Consulting Partner for dbt (“Data Build Tool”) Canonical: https://rittmananalytics.com/blog/2019/07/07/rittman-analytics-is-now-a-uk-consulting-partner-for-dbt-data-build-tool URL: https://rittmananalytics.com/blog/2019/07/07/rittman-analytics-is-now-a-uk-consulting-partner-for-dbt-data-build-tool Description: We’re very pleased to announce that Rittman Analytics is now an official Consulting Partner for dbt, working with our clients, the community and the wider dbt ecosystem to get the… Author: Mark Rittman Published: 2019-07-07T23:23:20+01:00 Modified: 2019-07-07T23:23:20+01:00 --- We’re very pleased to announce that [Rittman Analytics is now an official Consulting Partner for dbt](https://www.getdbt.com/ecosystem/), working with our clients, the community and the wider dbt ecosystem to get the most out of this open-source analytics framework and accompanying commercial dbtCloud service. You can read more about the role of Consulting Partners on the [new Ecosystem page on the getdbt website](https://www.getdbt.com/ecosystem/) and while you’re there, check-out [comments from real-world users of dbt](https://www.getdbt.com/users/) including one of ours, an excerpt from our recent blog post on *[How Rittman Analytics does Analytics](https://rittmananalytics.com/blog/2019/5/7/how-rittman-analytics)*. In that blog post we talked about how Rittman Analytics used dbt as part of our modular “extract, transform and load” data loading approach when preparing data ready for use with Looker: Since launching the company last year we’ve built-up a fair bit of experience implementing dbt and dbtCloud on several clients projects in areas such as - transforming event-based (Segment, mParticle), data pipeline (Stitch, Fivetran) and batch-loaded raw data sourced from SaaS, Telco, web application and other data sources - using Snowflake Data Warehouse, Postgres and Google BigQuery as sources/targets - integrating with GitHub, AWS CodeCommit for local/remote branch-based git development - creation of automated CI/CD dbt build test pipelines using dbtCloud and AWS CodeBuild/CodePipeline If you’re new to dbt or interested in how we use dbt and dbtCloud both internally and on client projects, check out our three recent blog posts on this topic: - [How Rittman Analytics does Analytics: Modern BI Stack Operational Analytics using Looker, Stitch, dbt and Google BigQuery](https://rittmananalytics.com/blog/2019/5/7/how-rittman-analytics) - [Customer Journey and Lifetime Value Analytics using Looker, Google BigQuery, Stitch and dbt](https://rittmananalytics.com/blog/2019/5/22/0r1fgtifyovghse903ha3vwdwwbp7j) - [Continuous Integration and Automated Build Testing with dbtCloud](https://rittmananalytics.com/blog/2019/6/11/continuous-integration-feature-branches-and-automated-build-tests-using-dbtcloud) and conversations we’ve had in the past with Tristan Handy, CEO and co-founder of Fishtown Analytics (primary sponsors of dbt) on our Drill to Detail Podcast: - [Drill to Detail Ep.69 ‘Looker, Tableau and Consolidation in the BI Industry’ featuring Special Guests Tristan Handy and Stewart Bryson](https://drilltodetail.rittmananalytics.com/podcast/2019/6/23/drill-to-detail-ep69-looker-tableau-and-consolidation-in-the-bi-industry-featuring-special-guests-tristan-handy-and-stewart-bryson) - [Drill to Detail Ep.59 ‘Looker, dbt and Digital Analytics Today’ With Special Guests Tristan Handy and Stewart Bryson](https://drilltodetail.rittmananalytics.com/podcast/2018/10/2/drill-to-detail-ep59-looker-dbt-and-digital-analytics-today-with-special-guests-tristan-handy-and-stewart-bryson) - [Drill to Detail Ep.33 ‘Building Out Analytics Functions in Startups’ With Special Guest Tristan Handy](https://drilltodetail.rittmananalytics.com/podcast/2017/7/3/drill-to-detail-ep33-building-out-analytics-functions-in-startups-with-special-guest-tristan-handy) [Contact us now](https://meetings.hubspot.com/mark-rittman) or email [info@rittmananalytics.com](mailto:info@rittmananalytics.com) if you’d like any help, or advice on your dbt implementation – we’re UK-based with clients in London, northern Europe the USA. --- # News on the Second London Looker Developer Meetup, 10th July 2019 at GoCardless, London Canonical: https://rittmananalytics.com/blog/2019/07/01/news-on-the-second-london-looker-developer-meetup-10th-july-2019-at-gocardless-london URL: https://rittmananalytics.com/blog/2019/07/01/news-on-the-second-london-looker-developer-meetup-10th-july-2019-at-gocardless-london Description: View fullsize The second London Looker Developer Meetup is less than two weeks away now, running from 6pm – 9pm on Wednesday July 10th 2019 at the GoCardless offices in… Author: Mark Rittman Published: 2019-07-01T16:39:51+01:00 Modified: 2019-07-01T16:39:51+01:00 --- View fullsize The second [London Looker Developer Meetup](https://www.meetup.com/Looker-London/events/262006740/) is less than two weeks away now, running from 6pm – 9pm on Wednesday July 10th 2019 at the GoCardless offices in Goswell Road, London. Co-hosted by Mark Rittman and Jon Palmer from GoCardless, join us for an evening of chat, networking, presentations and Q&As around the Looker platform and the following agenda: - **6:00 – 6:30pm:** Registration, Food & Networking - **6:30 – 6:40pm:** Welcome and Introduction – Jon Palmer, Head of BI at GoCardless and Mark Rittman, CEO of Rittman Analytics - **6:40 – 7:00pm**: “Getting Started with Looker” – Baha Sahin, Onfido - **7:00 – 7:20pm:** “Balancing Liquidity in an On-Demand Staffing Marketplace” – Charles Armitage, Florence - **7:20 – 7:35pm**: Update from Looker – Dave Hughes and Zara Wells, Customer Success at Looker - **7:35 – 8:00pm**: Data Analytics Panel - **8:00 – 9:00pm**: Networking and close We’re particularly looking-forward to Charles Armitage’s presentation as Florence were [Rittman Analytics’ first ever UK client](https://rittmananalytics.com/customers-index/#florence) when we launched the company back last year, and we’ve continued to work with Charles and the team since our original engagement right through to their [recent successful Series A funding round](https://www.linkedin.com/feed/update/urn:li:activity:6551413153027104768/). The First London Looker meetup ended-up oversubscribed so make sure you [register now](https://www.meetup.com/Looker-London/events/262006740/) if you’re planning on attending, and I’ll be joined by the Rittman Analytics team on the night if you’d like to chat with one of us about your Looker implementation – drop me an email at [mark.rittman@rittmananalytics.com](mailto:mark.rittman@rittmananalytics.com) if you’d like to schedule a chat in-advance. --- # Rittman Analytics is now a Segment Certified Implementation Partner Canonical: https://rittmananalytics.com/blog/2019/06/12/rittman-analytics-is-now-a-segment-certified-implementation-partner URL: https://rittmananalytics.com/blog/2019/06/12/rittman-analytics-is-now-a-segment-certified-implementation-partner Description: In our recent blog posts on How Rittman Analytics does Analytics: Modern BI Stack Operational Analytics and Customer Journey and Lifetime Value Analyticswe talked about using services from our partners… Author: Mark Rittman Published: 2019-06-12T23:40:01+01:00 Modified: 2019-06-12T23:40:01+01:00 --- In our recent blog posts on **[How Rittman Analytics does Analytics: Modern BI Stack Operational Analytics](https://rittmananalytics.com/blog/2019/5/7/how-rittman-analytics) and [Customer Journey and Lifetime Value Analytics](https://rittmananalytics.com/blog/2019/5/22/0r1fgtifyovghse903ha3vwdwwbp7j)**we talked about using services from our partners at Stitch, Fivetran and [Segment](https://segment.com/) to connect and integrate data ready for analysis with Looker. So far we’ve worked with joint Segment/Looker/Rittman Analytics customers such as [Let’s Do This](https://rittmananalytics.com/customers-index/#lets-do-this) to build customer and product analytics platforms on-top of Segment’s rich, event-level behavioural datasets. Now we’re pleased to announce that Rittman Analytics is now a Certified Implementation Partner for Segment, giving us the ability implement [Segment Connections](https://segment.com/product/connections/) along with [Segment Personas](https://segment.com/product/personas/) (Customer Data Platform) and [Segment Protocols](https://segment.com/product/protocols/) (Data Quality) on client analytics and wider digital marketing / digital transformation projects. Look-out for more information on how we’re using Segment technology together with modern, modular cloud analytics services from Looker, Stitch, Fivetran, Snowflake and Google Cloud Platform over the coming weeks, and if we can help with any of your Segment implementation questions just get in-touch at [info@rittmananalytics.com](mailto:info@rittmananalytics.com). --- # Continuous Integration and Automated Build Testing with dbtCloud Canonical: https://rittmananalytics.com/blog/2019/06/10/continuous-integration-and-automated-build-testing-with-dbtcloud URL: https://rittmananalytics.com/blog/2019/06/10/continuous-integration-and-automated-build-testing-with-dbtcloud Description: If you read my recent blog post on how Rittman Analytics built our operational analytics platform running on Looker, Stitch, Google BigQuery and dbt, you’ll know that we built-out the… Author: Mark Rittman Published: 2019-06-11T00:43:49+01:00 Modified: 2019-06-11T00:43:49+01:00 --- If you read my recent blog post on [how Rittman Analytics built our operational analytics platform](https://rittmananalytics.com/blog/2019/5/7/how-rittman-analytics) running on Looker, Stitch, Google BigQuery and dbt, you’ll know that we built-out the underlying data infrastructure using a modular Extract, Transform and Load (ELT) design pattern, like this: We version-control our dbt development environment using [git and Github.com](https://github.com/rittmananalytics/ra_dw_dbt), and do all new development beyond simple bug fixes as [git feature branches](https://git-scm.com/book/en/v2/Git-Branching-Branches-in-a-Nutshell), giving us a development process for dbt that looked like this: 1\. We’d start by cloning dbt git repo master branch from github.com to the developer’s workstation, which also would have dbt installed locally along with the Google Cloud SDK so that they can connect to our development BigQuery dataset. 2\. Create a new, local git branch for the new feature using the git CLI or a tool such as Github desktop 3\. Develop the new feature locally using the developer’s install of dbt, committing any changes to the feature branch in that developer’s local git repo after checking that all dbt tests have run successfully. locals-imac:ra\_dw markrittman$ dbt run –models harvest\_time\_entries harvest\_invoices –target devRunning with dbt=0.13.1Found 50 models, 8 tests, 0 archives, 0 analyses, 109 macros, 0 operations, 0 seed files, 35 sources20:53:11 | Concurrency: 1 threads (target=’dev’)20:53:11 |20:53:11 | 1 of 2 START table model ra\_data\_warehouse\_dbt\_dev.harvest\_invoices.. \[RUN\]20:53:13 | 1 of 2 OK created table model ra\_data\_warehouse\_dbt\_dev.harvest\_invoices \[OK in 1.80s\]20:53:13 | 2 of 2 START table model ra\_data\_warehouse\_dbt\_dev.harvest\_time\_entries \[RUN\]20:53:15 | 2 of 2 OK created table model ra\_data\_warehouse\_dbt\_dev.harvest\_time\_entries \[OK in 1.70s\]20:53:15 |20:53:15 | Finished running 2 table models in 5.26s.Completed successfullyDone. PASS=2 ERROR=0 SKIP=0 TOTAL=2locals-imac:ra\_dw markrittman$ dbt test –models harvest\_time\_entries harvest\_invoices –target devRunning with dbt=0.13.1Found 50 models, 8 tests, 0 archives, 0 analyses, 109 macros, 0 operations, 0 seed files, 35 sources20:53:37 | Concurrency: 1 threads (target=’dev’)20:53:37 |20:53:37 | 1 of 4 START test not\_null\_harvest\_invoices\_id………………….. \[RUN\]20:53:39 | 1 of 4 PASS not\_null\_harvest\_invoices\_id……………………….. \[PASS in 1.50s\]20:53:39 | 2 of 4 START test not\_null\_harvest\_time\_entries\_id………………. \[RUN\]20:53:40 | 2 of 4 PASS not\_null\_harvest\_time\_entries\_id……………………. \[PASS in 0.89s\]20:53:40 | 3 of 4 START test unique\_harvest\_invoices\_id……………………. \[RUN\]20:53:41 | 3 of 4 PASS unique\_harvest\_invoices\_id…………………………. \[PASS in 1.08s\]20:53:41 | 4 of 4 START test unique\_harvest\_time\_entries\_id………………… \[RUN\]20:53:42 | 4 of 4 PASS unique\_harvest\_time\_entries\_id……………………… \[PASS in 0.83s\]20:53:42 |20:53:42 | Finished running 4 tests in 5.27s.Completed successfully 4\. All dbt transformations at this stage are being deployed to our development BigQuery dataset. 5\. When the feature was then ready for deployment to our production BigQuery dataset, the developer would then push the changes in their local branch to the remote git repo, creating that branch if it didn’t already exist. 6\. Then they’d create a pull request using Github Desktop and the Github web UI summarising the changes and new features added by the development branch. 7\. I’d then review the PR, try and work out if the changes were safe to merge into the master branch and then accept the pull reques, and then overnight our dbtCloud service would clone that development git repo master branch and attempt to deploy the new set of transformations to our production BigQuery dataset. And sometimes, if for whatever reason that feature branch hadn’t been properly build-tested, that scheduled overnight deployment would then fail. So having noticed the new [Build on Pull Request feature](https://docs.getdbt.com/docs/cloud-enabling-continuous-integration-with-github) that comes with paid versions of dbtCloud, we’ve upgraded from the free version to the $100/month basic paid version and added an automated “continuous integration” build test to our feature branch development process so that it now looks like this: To set this automated build test feature up, we first l[inked our Github account to dbtCloud](https://docs.getdbt.com/docs/cloud-installing-the-github-application) and then created a new dbtCloud job that [triggers when a user submits a pull request](https://docs.getdbt.com/docs/cloud-enabling-continuous-integration-with-github). 8\. Now, when I come to review the pull request for this feature branch, there’s an automated test added by dbtCloud that checks to see whether this new version of my dbt package deploys without errors. 9\. Looking at the details behind this automated build test, I can see that dbtCloud has created a dataset and runs through a full package deployment and test cycle, avoiding any risk of breaking our production dbt environment should that test deployment fail. 10\. Checking back at the status of the test build in Github, I can see that this test completed with no issues, and I’m therefore safe to merge the changes in this feature branch into our development master branch, ready for dbtCloud to pick-up these changes overnight and deploy them as scheduled into production. Docs on this recent dbtCloud feature are [online here](https://docs.getdbt.com/docs/cloud-enabling-continuous-integration-with-github), and you’ll need at least the basic $100/month paid version of dbtCloud for this feature to become available in the web UI. It’s also dependent on Github as the git hosting service, so not available as an option if you’re using Gitlab, BitBucket or CodeCommit. --- # Drill to Detail Ep.66 ‘ETL, Incorta and the Death of the Star Schema’ with Special Guest Matthew Halliday Canonical: https://rittmananalytics.com/blog/2019/05/27/drill-to-detail-ep-66-etl-incorta-and-the-death-of-the-star-schema-with-special-guest-matthew-halliday URL: https://rittmananalytics.com/blog/2019/05/27/drill-to-detail-ep-66-etl-incorta-and-the-death-of-the-star-schema-with-special-guest-matthew-halliday Description: Mark Rittman is joined by Matthew Halliday from Incorta in this latest episode of the Drill to Detail Podcast to talk about the challenge of ETL and analytics on complex… Author: Mark Rittman Published: 2019-05-28T00:03:53+01:00 Modified: 2019-05-28T00:03:53+01:00 --- Mark Rittman is joined by Matthew Halliday from [Incorta](https://incorta.com/) in this [latest episode of the Drill to Detail Podcast](https://drilltodetail.rittmananalytics.com/podcast/2019/5/27/drill-to-detail-ep67-etl-incorta-and-the-death-of-the-star-schema-with-special-guest-matthew-halliday) to talk about the challenge of ETL and analytics on complex relational OLTP data models, previous attempts to solve these problems with products such as Oracle Essbase and Oracle E-Business Suite Extensions for Oracle Endeca and how those experiences, and others, led to his current role as co-founder and VP of Products at Incorta. - [The Death of the Star Schema: 3 Key Innovations Driving the Rapid Demise](http://go.incorta.com/2018.12.12WebinarTheDeathoftheStarSchema_LandingPage.html) - [Accelerating Analytics with Direct Data Mapping](http://go.incorta.com/Direct-Data-Mapping.html?utm_medium=website-content&utm_source=Incorta&utm_term=na&utm_content=na&utm_campaign=2017-wp-accelerating-analytics) - [Accelerating Operational Reporting & Analytics for Oracle E-Business Suite (EBS)](http://go.incorta.com/19-05-WP-Accelerating-Analytics-for-Oracle-EBS.html?utm_medium=website-content&utm_source=incorta&utm_term=na&utm_content=na&utm_campaign=2019-accelerating-ebs-analytics) - [The Good, the Bad, and the Ugly of Extract Transform Load (ETL)](http://go.incorta.com/2018.10Website-ContentEbook-ETLTheGoodBadUgly_LandingPage.html?utm_medium=website-content&utm_source=incorta&utm_term=na&utm_content=na&utm_campaign=2018-ebook-good-bad-ugly-etl) - [E-Business Suite Extensions for Endeca: Technical Considerations](https://www.youtube.com/watch?v=wetrx82C5_U) - [The Pain of Operational Reporting Solutions for Oracle E-Business Suite (EBS)](https://incorta.com/blog/2019/the-pain-of-operational-reporting-solutions-for-oracle-e-business-suite-ebs) --- # Drill to Detail Ep.66 ‘ETL, Incorta and the Death of the Star Schema’ with Special Guest Matthew Halliday Canonical: https://rittmananalytics.com/blog/2019/05/27/2019-5-27-drill-to-detail-ep66-etl-incorta-and-the-death-of-the-star-schema-with-special-guest-matthew-halliday URL: https://rittmananalytics.com/blog/2019/05/27/2019-5-27-drill-to-detail-ep66-etl-incorta-and-the-death-of-the-star-schema-with-special-guest-matthew-halliday Description: Mark Rittman is joined by Matthew Halliday from Incorta in this latest episode of the Drill to Detail Podcast to talk about the challenge of ETL and analytics on complex… Author: Mark Rittman Published: 2019-05-27T23:03:53+00:00 Modified: 2019-05-27T23:03:53+00:00 --- Mark Rittman is joined by Matthew Halliday from [Incorta](https://incorta.com/) in this [latest episode of the Drill to Detail Podcast](https://drilltodetail.rittmananalytics.com/podcast/2019/5/27/drill-to-detail-ep67-etl-incorta-and-the-death-of-the-star-schema-with-special-guest-matthew-halliday) to talk about the challenge of ETL and analytics on complex relational OLTP data models, previous attempts to solve these problems with products such as Oracle Essbase and Oracle E-Business Suite Extensions for Oracle Endeca and how those experiences, and others, led to his current role as co-founder and VP of Products at Incorta. - [The Death of the Star Schema: 3 Key Innovations Driving the Rapid Demise](http://go.incorta.com/2018.12.12WebinarTheDeathoftheStarSchema_LandingPage.html) - [Accelerating Analytics with Direct Data Mapping](http://go.incorta.com/Direct-Data-Mapping.html?utm_medium=website-content&utm_source=Incorta&utm_term=na&utm_content=na&utm_campaign=2017-wp-accelerating-analytics) - [Accelerating Operational Reporting & Analytics for Oracle E-Business Suite (EBS)](http://go.incorta.com/19-05-WP-Accelerating-Analytics-for-Oracle-EBS.html?utm_medium=website-content&utm_source=incorta&utm_term=na&utm_content=na&utm_campaign=2019-accelerating-ebs-analytics) - [The Good, the Bad, and the Ugly of Extract Transform Load (ETL)](http://go.incorta.com/2018.10Website-ContentEbook-ETLTheGoodBadUgly_LandingPage.html?utm_medium=website-content&utm_source=incorta&utm_term=na&utm_content=na&utm_campaign=2018-ebook-good-bad-ugly-etl) - [E-Business Suite Extensions for Endeca: Technical Considerations](https://www.youtube.com/watch?v=wetrx82C5_U) - [The Pain of Operational Reporting Solutions for Oracle E-Business Suite (EBS)](https://incorta.com/blog/2019/the-pain-of-operational-reporting-solutions-for-oracle-e-business-suite-ebs) --- # Customer Journey and Lifetime Value Analytics using Looker, Google BigQuery, Stitch and dbt Canonical: https://rittmananalytics.com/blog/2019/05/21/customer-journey-and-lifetime-value-analytics-using-looker-google-bigquery-stitch-and-dbt URL: https://rittmananalytics.com/blog/2019/05/21/customer-journey-and-lifetime-value-analytics-using-looker-google-bigquery-stitch-and-dbt Description: One of the most common analytics use-cases I come across on client projects is to understand the “customer journey”; the series of stages prospects and customers typically go through from… Author: Mark Rittman Published: 2019-05-22T00:13:17+01:00 Modified: 2026-07-02T00:30:56+01:00 --- ![Customer Journey and Lifetime Value Analytics using Looker, Google BigQuery, Stitch and dbt](https://cdn-images-1.medium.com/max/800/0*4E-2HBQReeVQhpJu) One of the most common analytics use-cases I come across on client projects is to understand the [“customer journey”](https://en.wikipedia.org/wiki/Customer_experience#Digital_customer_journey); the series of stages prospects and customers typically go through from becoming aware of a company’s products and services through to consideration, conversion, up-sell and repurchase and eventually, to churn. Customers typically interact with multiple digital and offline channels over their overall lifetime, and if we take the model of a consulting services business such as [Rittman Analytics](https://rittmananalytics.com/), a typical customer journey and use of those touch-points might look like the diagram below. ![](https://cdn-images-1.medium.com/max/800/0*4E-2HBQReeVQhpJu) Each customer and prospect interaction with those touch-points sends signals about their buying intentions, product and service preferences and readiness to move through to the next stage in the purchase funnel. ![](https://cdn-images-1.medium.com/max/800/0*NEGQX2ckdD0lkrKy) Creating visualization such as these in Looker typically involves two steps of data transformation either through a set of Looker (persistent) derived tables, or as we’ve done for our own operational analytics platform, through two [dbt](https://www.getdbt.com/) (Data Build Tool) models sequenced together as a transformation graph. To start, we select a common set of columns from each digital touchpoint channel as brought in by our project data pipeline tool, [Stitch](https://www.stitchdata.com/); Hubspot for inbound and outbound marketing communctions together with sales activity and closes/loses; Harvest for billable and non-billable client days; Segment for website visits and so on. For each source we turn each type of customer activity they record into a type of event, details of the event and a monetary value (revenue, cost etc) that we can then use to measure the overall lifetime value of each client later on. ![](https://cdn-images-1.medium.com/max/800/0*_Imn4FqJZCWRa4LG) Note the expressions at the start of the model definition that define a number of recency measures (days since last billable day, last contact day and so on) along with another set that are used for defining customer cohorts and bucketing activity into months and weeks since those first engagement date. {{ config( materialized='table', partition\_by='DATE(event\_ts)' ) }} SELECT \*, {{ dbt\_utils.datediff('last\_billable\_day\_ts', current\_timestamp(), 'day')}} AS days\_since\_last\_billable\_day, {{ dbt\_utils.datediff('last\_incoming\_email\_ts', current\_timestamp(), 'day')}} AS days\_since\_last\_incoming\_email, {{ dbt\_utils.datediff('last\_outgoing\_email\_ts', current\_timestamp(), 'day')}} AS days\_since\_last\_outgoing\_email, {{ dbt\_utils.datediff('last\_site\_visit\_day\_ts', current\_timestamp(), 'month')}} AS months\_since\_last\_site\_visit\_day, {{ dbt\_utils.datediff('last\_site\_visit\_day\_ts', current\_timestamp(), 'week')}} AS weeks\_since\_last\_site\_visit\_day FROM (SELECT \*, ROW\_NUMBER() OVER (PARTITION BY customer\_id ORDER BY event\_ts) AS event\_seq, MIN(CASE WHEN event\_type = 'Billable Day' THEN event\_ts END) {{ customer\_window\_over('customer\_id', 'event\_ts', 'ASC') }} AS first\_billable\_day\_ts, MAX(CASE WHEN event\_type = 'Billable Day' THEN event\_ts END) {{ customer\_window\_over('customer\_id', 'event\_ts', 'ASC') }} AS last\_billable\_day\_ts, MIN(CASE WHEN event\_type = 'Client Invoiced' THEN event\_ts END) {{ customer\_window\_over('customer\_id', 'event\_ts', 'ASC') }} AS first\_invoice\_day\_ts, MAX(CASE WHEN event\_type = 'Client Invoiced' THEN event\_ts END) {{ customer\_window\_over('customer\_id', 'event\_ts', 'ASC') }} AS last\_invoice\_day\_ts, MAX(CASE WHEN event\_type = 'Site Visited' THEN event\_ts END) {{ customer\_window\_over('customer\_id', 'event\_ts', 'ASC') }} AS last\_site\_visit\_day\_ts, MAX(CASE WHEN event\_type = 'Incoming Email' THEN event\_ts END) {{ customer\_window\_over('customer\_id', 'event\_ts', 'ASC') }} AS last\_incoming\_email\_ts, MAX(CASE WHEN event\_type = 'Outgoing Email' THEN event\_ts END) {{ customer\_window\_over('customer\_id', 'event\_ts', 'ASC') }} AS last\_outgoing\_email\_ts, MIN(event\_ts) {{ customer\_window\_over('customer\_id', 'event\_ts', 'ASC') }} AS first\_contact\_ts, DATE\_DIFF(date(event\_ts),MIN(CASE WHEN event\_type = 'Billable Day' THEN date(event\_ts) END) {{ customer\_window\_over('customer\_id', 'event\_ts', 'ASC') }},MONTH) AS months\_since\_first\_billable\_day, DATE\_DIFF(date(event\_ts),MIN(CASE WHEN event\_type = 'Billable Day' THEN date(event\_ts) END) {{ customer\_window\_over('customer\_id', 'event\_ts', 'ASC') }},WEEK) AS weeks\_since\_first\_billable\_day, DATE\_DIFF(date(event\_ts),MIN(CASE WHEN event\_type like '%Email%' THEN date(event\_ts) END) {{ customer\_window\_over('customer\_id', 'event\_ts', 'ASC') }},MONTH) AS months\_since\_first\_contact\_day, DATE\_DIFF(date(event\_ts),MIN(CASE WHEN event\_type like '%Email%' THEN date(event\_ts) END) {{ customer\_window\_over('customer\_id', 'event\_ts', 'ASC') }},WEEK) AS weeks\_since\_first\_contact\_day, MAX(CASE WHEN event\_type = 'Billable Day' THEN true ELSE false END) {{ customer\_window\_over('customer\_id', 'event\_ts', 'ASC') }} AS billable\_client, MAX(CASE WHEN event\_type LIKE '%Sales%' THEN true ELSE false END) {{ customer\_window\_over('customer\_id', 'event\_ts', 'ASC') }} AS sales\_prospect, MAX(CASE WHEN event\_type LIKE '%Site Visited%' THEN true ELSE false END) {{ customer\_window\_over('customer\_id', 'event\_ts', 'ASC') }} AS site\_visitor, MAX(CASE WHEN event\_details LIKE '%Blog%' THEN true ELSE false END) {{ customer\_window\_over('customer\_id', 'event\_ts', 'ASC') }} AS blog\_reader, MAX(CASE WHEN event\_type LIKE '%Podcast%' THEN true ELSE false END) {{ customer\_window\_over('customer\_id', 'event\_ts', 'ASC') }} AS podcast\_reader FROM -- sales opportunity stages (SELECT deals.lastmodifieddate AS event\_ts, customer\_master.customer\_id AS customer\_id, customer\_master.customer\_name AS customer\_name, deals.dealname AS event\_details, deals.dealstage AS event\_type, AVG(deals.amount) AS event\_value FROM {{ ref('customer\_master') }} AS customer\_master LEFT JOIN {{ ref('deals') }} AS deals ON customer\_master.hubspot\_company\_id = deals.associatedcompanyids LEFT JOIN {{ ref('hubspot\_owners') }} AS owners ON deals.hubspot\_owner\_id = CAST(owners.ownerid AS STRING) WHERE deals.lastmodifieddate IS NOT null {{ dbt\_utils.group\_by(n=5) }} UNION ALL -- consulting days SELECT time\_entries.spent\_date AS event\_ts, customer\_master.customer\_id AS customer\_id, customer\_master.customer\_name AS customer\_name, projects.name AS event\_details, CASE WHEN time\_entries.billable THEN 'Billable Day' ELSE 'Non-Billable Day' END AS event\_type, time\_entries.hours \* time\_entries.billable\_rate AS event\_value FROM {{ ref('customer\_master') }} AS customer\_master LEFT JOIN {{ ref('harvest\_projects') }} AS projects ON customer\_master.harvest\_customer\_id = projects.client\_id LEFT JOIN {{ ref('harvest\_time\_entries') }} AS time\_entries ON time\_entries.project\_id = projects.id WHERE time\_entries.spent\_date IS NOT null {{ dbt\_utils.group\_by(n=6) }} UNION ALL -- incoming and outgoing emails SELECT communications.communication\_timestamp AS event\_ts, customer\_master.customer\_id AS customer\_id, customer\_master.customer\_name AS customer\_name, communications.communications\_subject AS event\_details, CASE WHEN communications.communication\_type = 'INCOMING\_EMAIL' THEN 'Incoming Email' WHEN communications.communication\_type = 'EMAIL' THEN 'Outgoing Email' ELSE communications.communications\_subject END AS event\_type, 1 AS event\_value FROM {{ ref('customer\_master') }} AS customer\_master LEFT JOIN {{ ref('communications') }} AS communications ON customer\_master.hubspot\_company\_id = communications.hubspot\_company\_id WHERE communications.communication\_timestamp IS NOT null {{ dbt\_utils.group\_by(n=5) }} UNION ALL -- sales opportunity stages SELECT invoices.issue\_date AS event\_ts, customer\_master.customer\_id AS customer\_id, customer\_master.customer\_name AS customer\_name, invoices.subject AS event\_details, 'Client Invoiced' AS event\_type, SUM(invoices.amount) AS event\_value FROM {{ ref('customer\_master') }} AS customer\_master LEFT JOIN {{ ref('harvest\_invoices') }} AS invoices ON customer\_master.harvest\_customer\_id = invoices.client\_id WHERE invoices.issue\_date IS NOT null {{ dbt\_utils.group\_by(n=5) }} UNION ALL SELECT invoices.issue\_date AS event\_ts, customer\_master.customer\_id AS customer\_id, customer\_master.customer\_name AS customer\_name, invoice\_line\_items.description AS event\_details, 'Client Credited' AS event\_type, COALESCE(SUM(invoice\_line\_items.amount ), 0) AS event\_value FROM {{ ref('customer\_master') }} AS customer\_master LEFT JOIN {{ ref('harvest\_invoices') }} AS invoices ON customer\_master.harvest\_customer\_id = invoices.client\_id LEFT JOIN {{ ref('harvest\_invoice\_line\_items') }} AS invoice\_line\_items ON invoices.id = invoice\_line\_items.invoice\_id {{ dbt\_utils.group\_by(n=5) }} HAVING (COALESCE(SUM(invoice\_line\_items.amount ), 0) < 0) UNION ALL SELECT pageviews.timestamp AS event\_ts, customer\_master.customer\_id AS customer\_id, customer\_master.customer\_name AS customer\_name, pageviews.page\_subcategory AS event\_details, 'Site Visited' AS event\_type, sum(1) as event\_value FROM {{ ref('customer\_master') }} AS customer\_master LEFT JOIN {{ ref('pageviews') }} AS pageviews ON customer\_master.customer\_name = pageviews.network {{ dbt\_utils.group\_by(n=5) }} UNION ALL SELECT \* FROM (SELECT invoices.paid\_at AS event\_ts, customer\_master.customer\_id AS customer\_id, customer\_master.customer\_name AS customer\_name, invoices.subject AS event\_details, CASE WHEN invoices.paid\_at <\= invoices.due\_date THEN 'Client Paid' ELSE 'Client Paid Late' END AS event\_type, SUM(invoices.amount) AS event\_value FROM {{ ref('customer\_master') }} AS customer\_master LEFT JOIN {{ ref('harvest\_invoices') }} AS invoices ON customer\_master.harvest\_customer\_id = invoices.client\_id WHERE invoices.paid\_at IS NOT null {{ dbt\_utils.group\_by(n=5) }} ) ) WHERE customer\_name NOT IN ('Rittman Analytics', 'MJR Analytics') ) Deploying this model and then querying the resulting table, filtering on just a single customer and ordering rows by timestamp, you can see the history of all interactions by that customer with our sales and delivery channels. ![](https://cdn-images-1.medium.com/max/800/0*cR7SmHEjK3aryKmB) You can also plot all of these activity types into a combined area and bar chart, for example, to show billable days initially growing from zero to a level that we maintain over several months, and with events such as client being invoiced, incoming and outgoing communications and the client actually paying in-full at the end. ![](https://cdn-images-1.medium.com/max/800/0*C8qUql3ghozG0u0N) We can also do things such as restrict the event type reported on to billable days, and then use Looker’s dimension group and timespans feature to split the last set of weeks into days and week numbers to then show client utilisation over that period. ![](https://cdn-images-1.medium.com/max/800/0*3Co1garyusvs1oT6) Of course we can also take those monetary values assigned to events and use them to calculate the overall revenue contribution for this customer, factoring in costs incurred around preparing sales proposals, answering support questions and investing non-billable days in getting a project over the line. ![](https://cdn-images-1.medium.com/max/800/0*9vPEX014iu3F2-lq) Our second dbt model takes this event data and pivots each customer’s events into a series of separate columns, one for the first event in sequence, another for the second, another for the third and so on. ![](https://cdn-images-1.medium.com/max/800/0*QTF-JvMRZPutEaRw) Again, code for this dbt model is available in our [project git repo](https://github.com/rittmananalytics/ra_dw_dbt). {{ config( materialized='table' ) }} WITH event\_type\_seq\_final AS ( SELECT customer\_id, user\_session\_id AS event\_type\_seq, event\_type, event\_ts, is\_new\_session FROM (SELECT customer\_id, event\_type, last\_event, event\_ts, is\_new\_session, SUM(is\_new\_session) OVER (ORDER BY customer\_id ASC, event\_ts ASC ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS global\_session\_id, SUM(is\_new\_session) {{ customer\_window\_over('customer\_id', 'event\_ts', 'ASC') }} AS user\_session\_id FROM (SELECT \*, CASE WHEN event\_type != LAG(event\_type, 1) OVER (PARTITION BY customer\_id ORDER BY event\_ts ASC) OR last\_event IS NULL THEN 1 ELSE 0 END AS is\_new\_session FROM (SELECT customer\_id, customer\_name, CASE WHEN event\_type LIKE '%Email%' THEN 'Presales' WHEN event\_type LIKE '%Bill%' THEN 'Delivery' ELSE event\_type END AS event\_type, event\_ts, LAG(CASE WHEN event\_type LIKE '%Email%' THEN 'Presales' WHEN event\_type LIKE '%Bill%' THEN 'Delivery' ELSE event\_type END, 1) OVER (PARTITION BY customer\_id ORDER BY event\_ts ASC) AS last\_event FROM {{ ref('customer\_events') }} ORDER BY customer\_id ASC, event\_ts ASC, event\_type ASC ) last ORDER BY customer\_id ASC, event\_ts ASC, event\_type ASC ) final ORDER BY customer\_id ASC, is\_new\_session DESC, event\_ts ASC ) WHERE is\_new\_session = 1 {{ dbt\_utils.group\_by(n=5) }} ORDER BY customer\_id, user\_session\_id, event\_ts ) SELECT customer\_id, MAX(event\_type\_1) AS event\_type\_1, MAX(event\_type\_2) AS event\_type\_2, MAX(event\_type\_3) AS event\_type\_3, MAX(event\_type\_4) AS event\_type\_4, MAX(event\_type\_5) AS event\_type\_5, MAX(event\_type\_6) AS event\_type\_6, MAX(event\_type\_7) AS event\_type\_7, MAX(event\_type\_8) AS event\_type\_8, MAX(event\_type\_9) AS event\_type\_9, MAX(event\_type\_10) AS event\_type\_10 FROM (SELECT customer\_id, CASE WHEN event\_type\_seq = 1 THEN event\_type END AS event\_type\_1, CASE WHEN event\_type\_seq = 2 THEN event\_type END AS event\_type\_2, CASE WHEN event\_type\_seq = 3 THEN event\_type END AS event\_type\_3, CASE WHEN event\_type\_seq = 4 THEN event\_type END AS event\_type\_4, CASE WHEN event\_type\_seq = 5 THEN event\_type END AS event\_type\_5, CASE WHEN event\_type\_seq = 6 THEN event\_type END AS event\_type\_6, CASE WHEN event\_type\_seq = 7 THEN event\_type END AS event\_type\_7, CASE WHEN event\_type\_seq = 8 THEN event\_type END AS event\_type\_8, CASE WHEN event\_type\_seq = 9 THEN event\_type END AS event\_type\_9, CASE WHEN event\_type\_seq = 10 THEN event\_type END AS event\_type\_10 FROM event\_type\_seq\_final ) {{ dbt\_utils.group\_by(n=1) }} Pivoting each customer’s events in this way makes it possible then to visualize the first n events for a given set of customers using Looker’s [Sankey custom visualization](https://looker.com/platform/blocks/viz/sankey), as shown in the final screenshot below. ![](https://cdn-images-1.medium.com/max/800/0*Yi5OQ5p8OEAIkFpt) More details on our own internal use of Looker, Stitch, dbt and Google BigQuery to create a [modern operational analytics platform](https://rittmananalytics.com/blog/2019/5/7/how-rittman-analytics) can be found in this earlier blog post, and if you’re interested in how we might help you get started with Looker, check out our [Services page](https://rittmananalytics.com/services-index#looker). --- # How Rittman Analytics does Analytics: Modern BI Stack Operational Analytics using Looker, Stitch… Canonical: https://rittmananalytics.com/blog/2019/05/07/how-rittman-analytics-does-analytics-modern-bi-stack-operational-analytics-using-looker-stitch URL: https://rittmananalytics.com/blog/2019/05/07/how-rittman-analytics-does-analytics-modern-bi-stack-operational-analytics-using-looker-stitch Description: (First Published on the Rittman Analytics Blog, 8th May 2019) As well as creating modern BI stack analytics solutions for clients such as Florence, Let’s Do This and Colourpop using… Author: Mark Rittman Published: 2019-05-07T23:34:49+01:00 Modified: 2026-07-02T00:31:02+01:00 --- ![How Rittman Analytics does Analytics: Modern BI Stack Operational Analytics using Looker, Stitch…](https://cdn-images-1.medium.com/max/800/0*1Nro_nIXIFVojmsJ) *(First Published on the Rittman Analytics Blog,* [*8th May 2019*](https://rittmananalytics.com/blog/2019/5/7/how-rittman-analytics)*)* As well as creating [modern BI stack analytics solutions](https://www.google.com/search?q=modern+analytics+stack&oq=modern+analytics&aqs=chrome.0.69i59j69i60l2j69i57j69i60j0.2216j0j9&sourceid=chrome&ie=UTF-8) for clients such as [Florence](https://rittmananalytics.com/customers-index/#florence), [Let’s Do This](https://rittmananalytics.com/customers-index/#lets-do-this) and [Colourpop](https://rittmananalytics.com/customers-index/#colourpop) using technologies from our partners at [Looker](https://looker.com/), [Stitch](https://www.stitchdata.com/), [Segment](https://segment.com/), [Fivetran](https://fivetran.com/) and [Snowflake](https://www.snowflake.com/), it’s probably no surprise that we’ve used those same technologies and project experience to build-out our own internal analytics platform, covering data and operational analytics use-cases such as - Showing high-level KPIs that show progress towards our four main business objectives - Providing actionable context for these high-level KPIs and the ability to drill-into the detail behind them - Enabling ad-hoc querying and analysis of our complete dataset by technical and non-technical users - Providing a trusted, integrated analysis-ready data platform for other internal projects ![](https://cdn-images-1.medium.com/max/800/0*1Nro_nIXIFVojmsJ) I thought it might be of interest to go through how we’ve used the tools and techniques we use for client projects but in this case to deliver an analytics solution for our own internal use, sharing some of our thinking around how we defined our underlying KPI framework, the tools and design patterns we used when building out the data management part of the platform, and some of the front-end components and data visualizations we’ve put together using the Looker BI tool. ### Defining your KPI Framework When creating any sort of analytics solution the first thing you need to understand is — what are your business objectives; what do you need to measure to understand your progress towards those objectives; and how do you then define success? Key Performance Indicators (KPIs) and measures within our analytics platform are based around what’s needed to measure our progress towards four business objectives we set ourselves for FY2019: 1. Increase Billing Revenue 2. Increase Profitability 3. Increase customer retention, and 4. Increase our operational and delivery efficiency In-practice, we “certify” data around these four primary KPIs and also a number of secondary measures, with data in the platform normally up-to-date as of the previous business day and with historical data going back to when we started the business. ![](https://cdn-images-1.medium.com/max/800/0*cYl6LMlz0b7wgEKA) Other metrics, measures and dimension attributes are provided alongside these certified KPIs and measures and are considered “best endeavours” in terms of our certifying their accuracy, and users then of course able to create their own calculations for use in reports and analyses outside of this formal curation. ### Platform Architecture At a high-level our internal analytics platform extracts data on a regular basis from the SaaS-based services we use to run our business operations, joins that data together so that each dataset uses the same definition of clients, products and other reference data, then makes that combined dataset available for analysis as a set of business KPIs and an ad-hoc query environment. ![](https://cdn-images-1.medium.com/max/800/0*gYP56CalNhCgLb6f) The design of our internal analytics platform was based around a modular BI technology stack, ELT (“Extract, Land and Transform”) data warehouse loading pattern and modern software development approach that we use on all of our client analytics development engagements. ### Data Pipeline and Source Connectivity We’re digital-first, paperless and run our business operations using cloud-hosted software services such as: - [Harvest](https://rittman.harvestapp.com/) for timesheets, project invoicing and client-rechargeable expenses - [Harvest Forecast](https://forecastapp.com/1053510/) for project resource planning and revenue forecasts - [Xero](https://login.xero.com/) for accounting and payroll - [Hubspot](https://www.hubspot.com/) CRM for sales and contact management - [Docusign](https://account.docusign.com/) for contract management - [Jira](https://mjr-analytics.atlassian.net/) for project management and issue tracking - [BambooHR](https://rittman.bamboohr.co.uk/home/) for human resources and recruitment On the delivery side, Rittman Analytics partners with four different data pipeline-as-a-Service software vendors, each of which targets a different segment of the market and that segment’s particular use-cases: - [Stitch](https://www.stitchdata.com/), who focus on small to mid-market startup customers with data engineers who value service extensibility and their land-and-expand pricing model ($100/month for up to five sources, including Hubspot and Xero certified and Harvest community-supported connectors) - [Fivetran](http://fivetran.com/), aimed more at mid-market to enterprise customers and the data analyst personas, with a more turnkey solution and increased levels of service and coverage of enterprise data sources - [Segment](https://www.segment.io/), more of an “enterprise service bus” for event-level digital data streams, connecting multiple event producers with downstream event consumers and offering additional features around customer journey mapping, personas and schema validation - [Supermetrics](https://www.supermetrics.com/), aimed at growth hackers and digital marketers with a focus on simple self-service setup, comprehensive data dictionaries and a comprehensive catalog of advertising, marketing and social data sources delivered through a Google Sheets add-in or more recently, a direct connector into Google BigQuery through the GCP Marketplace. We chose to use Stitch’s service for our data pipeline for this internal project due to their out-of-the-box ability to source data from Harvest and Harvest Forecast; functionally both Fivetran and Stitch would meet our needs other than support for these two particular sources, and we’ve used Fivetran extensively on other internal and client projects in the past. ![](https://cdn-images-1.medium.com/max/800/0*JX9M2UjHvuT23Tv8) We also use Segment’s event tracking service to track visitor activity on our website and land those raw tracking events also in our BigQuery project, and intend to make use of Segment’s Personas product in due course to enable us to build out 360-degree views of our clients, site visitors and prospects based on their interactions across all of our various digital touchpoints (web, social media, inbound and outbound lead generation, chat etc) ### Data Transformation, Data Integration and Orchestration We transform and integrate raw data that Stitch syncs from each of our SaaS application sources into an integrated, query-orientated data warehouse dataset using the open-source [dbt](https://www.getdbt.com/) toolkit. Once we’d decided to work with a data pipeline-as-a-service such as Stitch together with a SQL-based data management platform like Google BigQuery, the decision to transform and integrate our data via a series of SQL SELECT was the obvious next design choice; by using dbt and version-controlling our scripts in a [Github repository](https://github.com/rittmananalytics/ra_dw_dbt) we increased our productivity when developing these transformation, adhered to modern software design principles and avoided cut-and-paste scripting in-favour of templated, maintainable code. ![](https://cdn-images-1.medium.com/max/800/0*TboPADzSw1mcw0x6) Transformation steps in dbt are basically SQL SELECTs with references to metadata models replacing hard-coded table names and use of macros as reusable, often cross-database transformation components giving three main benefits over hand-written, author-specific code: - Increased analyst productivity through leveraging libraries of best-practice transformation components - More maintainable loading processes with concise logic that “doesn’t repeat yourself” (DRY) - Portable code that can be deployed on BigQuery today or Snowflake tomorrow if required For example, as shown in the example code snippet below. ``` SELECT * FROM ( SELECT harvest_user_id, a.id, allocation, project_id, start_date, end_date, {{ dbt_utils.datediff(start_date, end_date, 'day')}} +1 AS forecast_days, a._sdc_sequence , MAX(a._sdc_sequence) OVER (PARTITION BY a.id ORDER BY a._sdc_sequence RANGE BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING) AS latest_sdc_sequence FROM {{ ref('harvest_forecast_assignments') }} a INNER JOIN {{ ref('harvest_forecast_people') }} p ON person_id = p.id ) WHERE _sdc_sequence = latest_sdc_sequence ) ``` In addition to the open-source dbt toolkit we also use the commercial [dbtCloud](https://cloud.getdbt.com/#/accounts/970/dashboard/) service from our friends at [Fishtown Analytics](https://www.fishtownanalytics.com/), mainly in order to then be able to execute our transformation graph in the cloud, like this: ![](https://cdn-images-1.medium.com/max/800/0*cOjq4k-VexBJiSQt) dbtCloud also hosts the data dictionary automatically generated from the metadata in our dbt package, and diagrams the transformation graph and data lineage we’ve defined through the dependencies we created between each of the models in the package. ![](https://cdn-images-1.medium.com/max/800/0*lFcLeygRZr3bPrSQ) As well as standardising each data source and mapping each source’s set of customer references into a single, master customer lookup table that allows us to analyze operations from sales prospect through project delivery to invoicing and payment, we also turn every customer touchpoint into events recorded against a single view of each customer. We then assign a financial revenue or cost value to each event in the customer journey, calculate metrics such as time between interactions (to measure engagement), value delivered to the client and use that information to better understand and segment our clients so as to provide the advice and service they’re looking for at this particular stage in their lifecycle and relationship with our team. ![](https://cdn-images-1.medium.com/max/800/0*J8MDn2VX6MnW-d7K) ### Analytics and Data Visualization We then use [Looker](https://www.looker.com/) as our primary analytics tool over the data prepared by dbt and stored in Google BigQuery. Everyone’s homepage in Looker is set by default to show those four main operational KPIs with comparison to target and with the last six months’ trend history, providing the context when combined with Looker’s ability to go deep into the detail behind those headline numbers so that users can decide and action on what they’re seeing (note that I’ve altered and obfuscated our actual numbers in the next few screenshots) ![](https://cdn-images-1.medium.com/max/800/0*5fFX4UoOh0rQbJ6e) We’ve also created a single business operations-wide Looker explore that enables users to analyze and explore all aspects of our sales, delivery and financial relationship with clients, suppliers, partners and prospects. In the example below we’ve again using the event model and applying sequencing to those events bucketed by months since first billable day, to help us understand how the the demand decay curve looks for each cohort of clients we’ve taken on over the first year we’ve been operating. ![](https://cdn-images-1.medium.com/max/800/0*3ZSbjrmLvv-o2Ag3) ### Embedded Analytics using Powered by Looker We also link fields in our Looker explore back to the online services that can provide more background detail on a metric displayed on the screen, in the example below linking back to Harvest to see details of the project displayed in the explore data visualisation. ![](https://cdn-images-1.medium.com/max/800/0*RYBXK7y73r5s0y-M) Finally, for team members who are more likely to be working within our knowledge-sharing and internal company portal site, we use [Powered by Looker](https://looker.com/product/embedded-analytics) embedded analyics within [Notion](https://www.notion.so/), connecting our internal analytics metrics and insights directly into the productivity and workflow tools used by the team, bringing data and analytics to the problems they’re looking to solve rather than the other way around. ![](https://cdn-images-1.medium.com/max/800/0*scpU4znCiuismbtw) So, hopefully this look behind the scenes at how we use these modern analytics stack technologies for our own analytics needs has been useful, and feel free to leave comments or contact me at mark.rittman@rittmananalytics.com if any of this would be applicable to your business. In the meantime, the dbt and Looker git repos used to build-out our platform are available as public git repos on Github: - [https://github.com/rittmananalytics/ra\_dw\_dbt](https://github.com/rittmananalytics/ra_dw_dbt) - [https://github.com/rittmananalytics/ra\_consulting\_services\_dw](https://github.com/rittmananalytics/ra_consulting_services_dw) Feel free to fork, improve on what we’ve done, submit PRs or just see a bit more of the detail behind the examples in this post. --- # Supermetrics, Google BigQuery and Data Pipelines for Digital Marketers Canonical: https://rittmananalytics.com/blog/2019/04/14/supermetrics-google-bigquery-and-data-pipelines-for-digital-marketers URL: https://rittmananalytics.com/blog/2019/04/14/supermetrics-google-bigquery-and-data-pipelines-for-digital-marketers Description: Chances are that around the corner from the analytics project you’re working on that’s using SaaS data piped-into your data warehouse using a service from Stitch or FIvetran, there’s a… Author: Mark Rittman Published: 2019-04-14T23:45:46+01:00 Modified: 2019-04-14T23:45:46+01:00 --- Chances are that around the corner from the analytics project you’re working on that’s using SaaS data piped-into your data warehouse using a service from Stitch or FIvetran, there’s a marketing analyst using Google Sheets and an add-in from Supermetrics to analyze their channel advertising spend. Supermetrics are the biggest analytics tool vendor company you’ve never heard, and that’s because their products are aimed squarely at marketers, not data engineers or data analysts, and within that particular market they’re massive; what they sell is easy-to-setup connectivity to all the major advertising and social networks directly from within Google Sheets in the form of the [Supermetrics Google Sheets Add-In](https://supermetrics.com/product/google-sheets), a product we’ve used in the past when looking to to understand how well we place in Google search results for keywords relevant to our business. Supermetrics do two things well; they provide connectivity to all the major ad networks that marketers’ use (and there’s lots of them beyond the basic Adwords, Facebook Ads and Snapchat Ads that we’re most familiar with), and they focus exclusively on the non-technical, marketing user rather than data engineers and data analysts, providing in-addition to their Google Sheets add-in a [connector for Google Data Studio](https://supermetrics.com/product/data-studio), for example. But although you can use Google Sheets as federated sources for Google BigQuery tables and theoretically you could then use the scheduled data refresh feature in Supermetrics’ add-in to regularly download new data and thereby create a sort-of data pipeline into your warehouse, in-practice this probably requires a bit more knowledge of how BigQuery works than the average marketer is likely to have. So I was particularly interested to see that one of the new BigQuery features announced at Google Cloud Next’19 last week was a number of [Supermetrics-provided data connectors for BigQuery](https://supermetrics.com/blog/supermetrics-for-bigquery) in the Google Cloud Marketplace, providing data pipeline-as-a-service connectivity from ad network sources such as Bing Ads, LinkedIn ads, AdRoll and others and with data being fed directly into Google BigQuery using their Data Transfer service. Some of these sources are also covered by new connectors from Fivetran also in this same Google Cloud Marketplace, so what’s the difference between a data pipeline provided by Supermetrics compared to one from Fivetran or Stitch? Setting up a Supermetrics data pipeline using Google BigQuery’s data transfer service is pretty straightforward and not that different from setting up a data pipeline using Fivetran or Google’s own underlying data transfer technology, as shown in the screenshots below: Its when you look at how Supermetrics land data into BigQuery though that you see their difference in approach; instead of presenting you with several tables of data mapped to the structure of the API calls for each service, the Supermetrics BigQuery tables are typically denormalized (flattened into one big table of data) and come with extensive and verbose column descriptions and metadata, making it much easier for marketers to know what each column is used for and the precise definition of all the metrics and dimension attributes. Pricing is per data source with a 14-day free trial at the start, to me it looks like Supermetrics are looking to position their pipeline-as-a-service in-between Stitch and Fivetran’s in-terms of pricing but with the service aimed at marketers, not data engineers. Details of pricing and pricing metrics are on the [Supermetrics support page](https://support.supermetrics.com/support/home) and full details of the service are on the [Supermetrics for BigQuery](https://supermetrics.com/product/bigquery) product homepage. --- # Join us at Looker JOIN 2019, London on April 9th – The Brewery, 52 Chiswell Street Canonical: https://rittmananalytics.com/blog/2019/04/05/join-us-at-looker-join-2019-london-on-april-9th-the-brewery-52-chiswell-street URL: https://rittmananalytics.com/blog/2019/04/05/join-us-at-looker-join-2019-london-on-april-9th-the-brewery-52-chiswell-street Description: Looker JOIN London is coming to London next week and Rittman Analytics will be at the event as one of the Silver Sponsors – come and visit us at Stand… Author: Mark Rittman Published: 2019-04-05T14:58:19+01:00 Modified: 2019-04-05T14:58:19+01:00 --- [Looker JOIN London](https://looker.com/events/join-the-tour/london) is coming to London next week and Rittman Analytics will be at the event as one of the [Silver Sponsors](https://looker.com/events/join-the-tour/london#sponsors) – come and visit us at Stand 10, right at the front of the main hall just as you walk in! JOIN London is an exciting opportunity to meet with Looker experts, data mavericks and industry thought leaders, all on-hand to share proven insights on how to become more data-driven. Get ready for thought-provoking presentations, informative discussions and interactive sessions, adopted from the best of Looker’s JOIN conferences that originally ran in San Francisco, USA. We’ll be there all day and keen to hear about your data initiatives, project needs or just to have a chat about Looker, analytics and driving value from data. Registration is free and still open, check-out the [event details page](https://looker.com/events/join-the-tour/london#register) for more information. --- # Released Today: The M.O.M.E. Project – Multiple OLAP Machine Emulator Canonical: https://rittmananalytics.com/blog/2019/04/01/released-today-the-m-o-m-e-project-multiple-olap-machine-emulator URL: https://rittmananalytics.com/blog/2019/04/01/released-today-the-m-o-m-e-project-multiple-olap-machine-emulator Description: On this day last year we broke the news on Oracle Discoverer Cloud Service; this year we go one-better with the announcement of a new open-source software and homebrew hardware… Author: Mark Rittman Published: 2019-04-01T10:33:52+01:00 Modified: 2019-04-01T10:33:52+01:00 --- On this day last year we [broke the news](https://twitter.com/markrittman/status/980422422330560512) on Oracle Discoverer Cloud Service; this year we go one-better with the announcement of a new open-source software and homebrew hardware project now hitting its 1.0.0 release milestone: [M.O.M.E. – Multiple OLAP Machine Emulator](https://github.com/rittmananalytics/mome), sponsored by Rittman Analytics. From the [project homepage on Github](https://github.com/rittmananalytics/mome): “MOME is a framework for emulating multi-dimensional Online Analytical Processing (“MOLAP”) engines, 4GL programming languages and the servers and OLAP applications that ran on those engines. MOME’s purpose is to preserve decades of multi-dimensional MOLAP engine and software history. As analytical processing technology continues to rush forward, MOME prevents this important “vintage” technology from being lost and forgotten. This is achieved by documenting the MOLAP engine, its multi-dimensional storage format and 4GL query languages, along with the software and sample databases for those engines such as Microsoft OLAP Services 7.0 AdventureWorks. Over time, MOME (originally stood for Multiple OLAP Machine Emulator) absorbed the sister-project MESS (Multi-dimensional Engine Super System), so MOME now documents a wide variety of (mostly vintage) MOLAP, Relational OLAP (“ROLAP”) and now Hybrid OLAP “HOLAP” engines in addition to the MOLAP engines and desktop applications that were its initial focus.” --- # Released Today: The M.O.M.E. Project – Multiple OLAP Machine Emulator Canonical: https://rittmananalytics.com/blog/2019/04/01/2019-04-01-2019-4-1-released-today-the-mome-project-multiple-olap-machine-emulator URL: https://rittmananalytics.com/blog/2019/04/01/2019-04-01-2019-4-1-released-today-the-mome-project-multiple-olap-machine-emulator Description: On this day last year we broke the news on Oracle Discoverer Cloud Service; this year we go one-better with the announcement of a new open-source software and homebrew hardware… Author: Mark Rittman Published: 2019-04-01T09:33:52+00:00 Modified: 2019-04-01T09:33:52+00:00 --- On this day last year we [broke the news](https://twitter.com/markrittman/status/980422422330560512) on Oracle Discoverer Cloud Service; this year we go one-better with the announcement of a new open-source software and homebrew hardware project now hitting its 1.0.0 release milestone: [M.O.M.E. – Multiple OLAP Machine Emulator](https://github.com/rittmananalytics/mome), sponsored by Rittman Analytics. From the [project homepage on Github](https://github.com/rittmananalytics/mome): “MOME is a framework for emulating multi-dimensional Online Analytical Processing (“MOLAP”) engines, 4GL programming languages and the servers and OLAP applications that ran on those engines. MOME’s purpose is to preserve decades of multi-dimensional MOLAP engine and software history. As analytical processing technology continues to rush forward, MOME prevents this important “vintage” technology from being lost and forgotten. This is achieved by documenting the MOLAP engine, its multi-dimensional storage format and 4GL query languages, along with the software and sample databases for those engines such as Microsoft OLAP Services 7.0 AdventureWorks. Over time, MOME (originally stood for Multiple OLAP Machine Emulator) absorbed the sister-project MESS (Multi-dimensional Engine Super System), so MOME now documents a wide variety of (mostly vintage) MOLAP, Relational OLAP (“ROLAP”) and now Hybrid OLAP “HOLAP” engines in addition to the MOLAP engines and desktop applications that were its initial focus.” --- # MJR Analytics is now Rittman Analytics … and an Update on our First Six Months Canonical: https://rittmananalytics.com/blog/2019/03/11/mjr-analytics-is-now-rittman-analytics-and-an-update-on-our-first-six-months URL: https://rittmananalytics.com/blog/2019/03/11/mjr-analytics-is-now-rittman-analytics-and-an-update-on-our-first-six-months Description: As of last Friday our company name has changed from MJR Analytics to Rittman Analytics. Nothing else has changed in-terms of ownership or services we provide but we wanted to… Author: Mark Rittman Published: 2019-03-11T12:08:37+01:00 Modified: 2019-03-11T12:08:37+01:00 --- As of last Friday our company name has changed from MJR Analytics to Rittman Analytics. Nothing else has changed in-terms of ownership or services we provide but we wanted to take advantage of our ability now to put our name and reputation front-and-centre of our branding and website. All emails to our old @mjr-analytics.com domain will automatically redirect, and any links pointing to [https://www.mjr-analytics.com](https://www.mjr-analytics.com/) will automatically redirect to [http://rittmananalytics.com](http://rittmananalytics.com/). Other than changing our name, since our launch last September we’ve worked with a number of great customers together with our vendor partners, with five of those projects written-up as case studies and customer quotes in the [Customers section](https://rittmananalytics.com/customers-index/#lets-do-this) of our website: - [Colourpop](https://rittmananalytics.com/customers-index/#colourpop) : eCommerce and customer analytics using Shopify, Redshift and Looker - [Let’s Do This](https://rittmananalytics.com/customers-index/#lets-do-this) : eCommerce/conversion analytics using Segment, Redshift and Looker - [Florence](https://rittmananalytics.com/customers-index/#florence) : Operational analytics, retention and RFM segmentation using Looker and PostgreSQL - [INTO University Partnerships](https://rittmananalytics.com/customers-index/#into-university-partnerships) : Digital Analytics Strategy for MS PowerBI and Heap Analytics - [Qubit](https://mjr-analytics.com/customers-index/#qubit) : Data Engineering and customer/personalization analytics using Google BigQuery and Looker We’re also co-chaired and helped organize the recent [Looker London Dev Meetup](https://www.meetup.com/Looker-London/events/257933863/) together with Jon Palmer from GoCardless, and presented a short session on [Improving your User Engagement with Looker’s new System Activity Model](https://rittmananalytics.com/s/Improve-your-User-Engagement-with-Lookers-new-System-Activity-Model.pdf) – look out for the next Meetup in June later this year! > Kicking-off the London [@LookerData](https://twitter.com/LookerData?ref_src=twsrc%5Etfw) Developer Meet-up at [@GoCardless](https://twitter.com/GoCardless?ref_src=twsrc%5Etfw) with an intro from [@jonjamespalmer](https://twitter.com/jonjamespalmer?ref_src=twsrc%5Etfw) and first talk by [@DylanBakerCFM](https://twitter.com/DylanBakerCFM?ref_src=twsrc%5Etfw) [https://t.co/pa8wm6TcXu](https://t.co/pa8wm6TcXu) [pic.twitter.com/pdEVgk2uye](https://t.co/pdEVgk2uye) > > — Mark Rittman (@markrittman) [February 7, 2019](https://twitter.com/markrittman/status/1093576412349558784?ref_src=twsrc%5Etfw) [https://platform.twitter.com/widgets.js](https://platform.twitter.com/widgets.js) [Looker](https://www.looker.com/) is our primary, strategic vendor partner and since our launch last year we’ve also partnered with selected vendors that provide features complementary to Looker: - [Segment](https://www.segment.io/) for integrating event-level data sources, single customer view and data quality - [Fivetran](https://www.fivetran.com/) for no-ops data pipelines - [Qubit](https://www.qubit.com/) for consumer behavioural, segmentation and personalization analytics - [Snowflake](https://www.snowflake.com/) for no-ops cloud data warehousing-as-a-service - [Oracle](https://www.oracle.com/) for enterprise-level analytics and ERP/CRM integration We’re also now recruiting for our first [permanent consulting roles](https://rittman.bamboohr.co.uk/jobs/view.php?id=21) to work alongside myself and our team of experienced contract developers and data scientist, and we’re looking forward to being one of the sponsors for the upcoming [Looker Join 2019 event in London](https://looker.com/events/join-the-tour/london) – see some of you there if not sooner! --- # Drill to Detail Podcast Returns with Ep.60 ‘A Deeper Look Into Looker’ With Special Guest Lloyd Tabb Canonical: https://rittmananalytics.com/blog/2019/03/05/drill-to-detail-podcast-returns-with-ep-60-a-deeper-look-into-looker-with-special-guest-lloyd-tabb URL: https://rittmananalytics.com/blog/2019/03/05/drill-to-detail-podcast-returns-with-ep-60-a-deeper-look-into-looker-with-special-guest-lloyd-tabb Description: One of our CEO and founder Mark Rittman’s side projects is the Drill to Detail Podcast, a podcast series where Mark interviews a special guest each episode to talk about… Author: Mark Rittman Published: 2019-03-05T22:08:38+01:00 Modified: 2019-03-05T22:08:38+01:00 --- One of our CEO and founder Mark Rittman’s side projects is the [Drill to Detail Podcast](https://www.drilltodetail.com/), a podcast series where Mark interviews a special guest each episode to talk about their work, their products and their take on the analytics, big data and data management industry. Drill to Detail is now on its 60th episode and now starts its third season with Mark joined by Lloyd Tabb, Founder, CTO and Chairman of Looker to talk about the foundational story of Looker and LookML, query latency and semantic models, analytic engines and code IDEs, analytics developer workflows and the rise of cloud elastically-scalable databases, packaged applications and embedded analytics and why learning (and loving) are the long-term keys to analytics and business success. - [Looker JOIN: The Tour 2019](https://looker.com/events/join-the-tour) - [“7 Reasons Looker Built a New Language for Data”](https://looker.com/blog/7-reasons-looker-built-a-new-language-for-data) - [“How do you decide what to model in dbt vs LookML?”](https://blog.fishtownanalytics.com/how-do-you-decide-what-to-model-in-dbt-vs-lookml-dca4c79e2304) – Tristan Handy - [“Looker co-founder finds freedom and inspiration on his bike rides”](https://www.bizjournals.com/sanjose/news/2018/06/15/lloyd-tabb-looker.html) – BizJournals.com - [Looker Packaged Applications](https://looker.com/product/applications) - [“A simple explanation of Symmetric Aggregates or: Why On Earth Does My SQL Look Like That?”](https://discourse.looker.com/t/a-simple-explanation-of-symmetric-aggregates-or-why-on-earth-does-my-sql-look-like-that/6601) – Daniel Mintz - [Drill to Detail Ep.48 ‘Mondrian OLAP, Apache Calcite and Database Dis-Aggregation’ With Special Guest Julian Hyde](https://www.drilltodetail.com/podcast/2018/1/23/drill-to-detail-ep48-mondrian-olap-apache-calcite-and-database-dis-aggregation-with-special-guest-julian-hyde) --- # Looker London Meetup on Thursday, February 7th 2019 – Registration Now Open! Canonical: https://rittmananalytics.com/blog/2019/01/21/looker-london-meetup-on-thursday-february-7th-2019-registration-now-open URL: https://rittmananalytics.com/blog/2019/01/21/looker-london-meetup-on-thursday-february-7th-2019-registration-now-open Description: If you’re in London on Thursday 7th February 2019 and a developer or end-user working with the Looker Data Platform, we’re very pleased to be launching together with Jon Palmer… Author: Mark Rittman Published: 2019-01-21T01:38:17+01:00 Modified: 2019-01-21T01:38:17+01:00 --- If you’re in London on Thursday 7th February 2019 and a developer or end-user working with the Looker Data Platform, we’re very pleased to be launching together with [Jon Palmer](https://www.linkedin.com/in/jonathan-palmer-46ba6434/) from [Go Cardless](https://gocardless.com/) the first ever [Looker London Dev Meetup](https://www.meetup.com/Looker-London/events/257933863/), sponsored by the good folks at Looker and hosted at Go Cardless’ HQ at [65 Goswell Road, London EC1V 7EN](https://goo.gl/maps/4w1UN7kDLou). The aim is to create a space for showcasing and discussing what data people in London are building in and around the Looker ecosystem, and for this first meetup we have three great customer and partner speakers sharing their insights on Looker development and project implementations: - “Scale securely with the Looker API” Dylan Baker, Consultant - “Internal and External Multi-tenancy with Looker” – Ian Thompson, King - “Increase End-User Engagement with Looker 6 System Activity Model” – Mark Rittman, MJR Analytics followed by a panel session where the speakers are joined by one of our guests from Looker: - “Data Analytics Panel – Hear from Industry experts and Looker users from GoCardless, King, MJR analytics and Looker.” Registration and attendance is free but spaces are limited, so make sure you register soon if you’d like to come along – the meetup starts at at 5.30pm with drinks, food and introductions from Jon, full agenda is on the [meetup page](https://www.meetup.com/Looker-London/events/257933863/) and we’re aiming to wrap-up around 9pm with closing comments from me and a short survey on future meetup content. --- # MJR Analytics Sessions at UKOUG Tech’18, Liverpool ACC, 3rd-5th December 2018 Canonical: https://rittmananalytics.com/blog/2018/11/27/mjr-analytics-sessions-at-ukoug-tech18-liverpool-acc-3rd-5th-december-2018 URL: https://rittmananalytics.com/blog/2018/11/27/mjr-analytics-sessions-at-ukoug-tech18-liverpool-acc-3rd-5th-december-2018 Description: Oracle Openworld may have San Francisco and the Moscone Center but you can’t beat Liverpool and the wind whipping in-off the Mersey in early December, and so we’re very pleased… Author: Mark Rittman Published: 2018-11-27T14:54:20+01:00 Modified: 2018-11-27T14:54:20+01:00 --- Oracle Openworld may have San Francisco and the Moscone Center but you can’t beat Liverpool and the wind whipping in-off the Mersey in early December, and so we’re very pleased to be presenting once again at the [UK Oracle User Group Technology Conference & Exhibition](http://exhibition18.ukoug.org/home) running next week at the Liverpool ACC, December 3rd-5th 2018. Mark Rittman is presenting two sessions on Oracle’s new autonomous analytics and data warehousing platforms: - **[“Planning a Strategy for Autonomous Analytics & Data Warehouse Cloud”](https://www.ukougconferences.org.uk/ukoug/frontend/reg/absViewDocumentFE.csp?popup=1&documentID=873&eventID=2)**, Monday 3rd December 2:30 PM – 3:15 PM in Room 13 - ”As Oracle Analytics and Data Warehousing becomes self-driving and autonomous, the need for a strategy within your BI function becomes all the more important. How you deliver BI content to your users, the skills your developers now need and the most efficient way to manage your cloud estate are vital components of an autonomous cloud analytics strategy; this session will explain what’s changed, what’s significant and what are the implications of that change.By reference to today’s Oracle BI technology and applications and explaining how they now run autonomously as services in Oracle Cloud we’ll cover what are the key changes and its implications, what this means in terms of budgets and planning your investment, what are the key technology and process changes, what this means for team resourcing and roles – and just as importantly, what can be safely ignored for now.” - **[“BI Developer to Data Engineer with Oracle Analytics Cloud Data Lake Edition”](https://www.ukougconferences.org.uk/ukoug/frontend/reg/absViewDocumentFE.csp?popup=1&documentID=870&eventID=2)**, Tuesday 4th December 12:35 PM – 1:20 PM Room 11A - ”In this session, we’ll look at the role of the data engineer in designing, provisioning, and enabling an Oracle Cloud data lake using Oracle Analytics Cloud Data Lake Edition. We’ll also examine the use of data flow and data pipeline authoring tools and how machine learning and AI can be applied to this task. Furthermore, we’ll explore connecting to database and SaaS sources along with sources of external data via Oracle Data-as-a-Service. Finally we’ll delve into how traditional Oracle Analytics developers can transition their skills into this role and start working as data engineers on Oracle Public Cloud data lake projects.” Mark Rittman will be at the event all week, so if you’ve got any questions about the topics we’re talking about in the two sessions or any aspect of Oracle Analytics, big data or Autonomous Data Warehouse either [drop us an email](mailto:info@mjr-analytics.com) beforehand, give us a call on +44 7866 568246 or just stop us in the corridor and we’ll be pleased to chat then. --- # Five Thoughts About Thomas Kurian’s Move to become CEO of Google Cloud Platform Canonical: https://rittmananalytics.com/blog/2018/11/17/five-thoughts-about-thomas-kurians-move-to-become-ceo-of-google-cloud-platform URL: https://rittmananalytics.com/blog/2018/11/17/five-thoughts-about-thomas-kurians-move-to-become-ceo-of-google-cloud-platform Description: The news broke late on Friday that Thomas Kurian is joining Google to become their new CEO for Google Cloud Platform. Five thoughts on Kurian’s move: It’s a smart move… Author: Mark Rittman Published: 2018-11-17T18:09:59+01:00 Modified: 2018-11-17T18:09:59+01:00 --- [The news broke late on Friday](https://techcrunch.com/2018/11/16/former-oracle-exec-thomas-kurian-to-replace-diane-greene-as-head-of-google-cloud/) that Thomas Kurian is joining Google to become their new CEO for Google Cloud Platform. Five thoughts on Kurian’s move: 1. It’s a smart move made by an exceptionally smart guy. [Brave, as well](https://www.cnbc.com/2018/11/16/kurian-has-to-overcome-a-bitter-feud-between-google-and-oracle.html), given the recent history between Oracle and Google but also not surprising given his drive and presumably point to prove. I met him a few times as part of Oracle’s ACE Director program and he’s the only software exec I know who can talk long-term strategy and vision one minute and then explain to you in-detail how it all works, and doing it all with great humility and courtesy. 2. The fact that GCP is spoken-of as an also-ran at 10% market share [whilst Oracle Cloud gets bundled in with “Next 10”](https://techcrunch.com/2018/06/20/oracle-could-be-feeling-cloud-transition-growing-pains/) shows what a mountain Oracle have to climb to even become a contender to compete with Microsoft and Amazon in the cloud business – and their insistence on only allowing their SaaS and PaaS apps to run in Oracle Cloud is a worrying parallel with the [“Windows Tax” that Microsoft’s Office and Server products teams had to pay back in the Steve Ballmer Era](https://stratechery.com/2018/the-end-of-windows/), but with Oracle’s equivalent to Satya Nadella having lost the argument and jumped-ship 3. But Oracle will survive and this has happened many times before – Ray Lane, Charles Philips, Tom Seibel, Mark Benioff all left and in many cases founded massively successful and competitive businesses, client-server went to internet architecture and then internet went to cloud, it’s all part of how businesses renew and people move on and up, and there’s plenty more smart (and most likely, smarter) people left in Oracle and Larry Ellison is still just as driven, competitive and on-top of his game. 4. Look out for a very interesting (as in Brexit, interesting to watch but not to be part of) [culture clash](https://www.cnbc.com/2018/11/16/kurian-has-to-overcome-a-bitter-feud-between-google-and-oracle.html) at GCP, with TK about the most [top-down leader](https://www.bloomberg.com/news/articles/2018-11-16/google-cloud-ceo-diane-greene-to-leave-oracle-s-kurian-steps-in) of a product development team I’ve ever seen and Google, famously, [engineering-focused beanbag-friendly and bottom-up](https://neilpatel.com/blog/googles-culture-of-success/). Add to that the feelings most Googler’s have towards Oracle and TK will have his work cut-out early on to win the various teams over – my guess is that his work-ethic, technical chops and sheer ability will do just that and if this role is a success, Sergey and Larry may well have found their new [Eric Schmidt](https://en.wikipedia.org/wiki/Eric_Schmidt "Eric Schmidt") but this time with technical skills to match theirs – but there’s always the chance that culture will prevail and he’ll be the next [Marissa Meye](https://www.cnbc.com/2017/05/31/why-yahoo-ceo-marissa-mayer-is-the-least-likable-ceo-in-tech.html)r instead. No pressure there then.. 5. Expect to see GCP moving increasingly into packaged SaaS application and analytics solutions for ERP, CRM and Financials to complement their commoditised IaaS and PaaS cloud business and leveraging their massive GSuite and increasingly ChromeOS install base. Just think what they could do if they had access to all the world’s structured business transactional data as well as the unstructured search, email and spreadsheet data they have now – even more comprehensive and market-leading search, the world’s biggest and most valuable ML training set, and a business model that could provide these applications and packaged analytics for free in exchange for exclusive access to that data and Google being your default search engine. That’s the real existential threat to Oracle; spending all their time trying to win an un-winnable cloud infrastructure war and then GCP coming along and making ERP, CRM and business applications essentially free. --- # Slides and Forbes.com Article from “Data Warehouse Like a Tech Startup with Oracle Autonomous Data Warehouse Cloud” Canonical: https://rittmananalytics.com/blog/2018/11/14/slides-and-forbes-com-article-from-data-warehouse-like-a-tech-startup-with-oracle-autonomous-data-warehouse-cloud URL: https://rittmananalytics.com/blog/2018/11/14/slides-and-forbes-com-article-from-data-warehouse-like-a-tech-startup-with-oracle-autonomous-data-warehouse-cloud Description: One of the sessions we delivered at Oracle Openworld 2018 was “Data Warehouse Like a Tech Startup with Oracle Autonomous Data Warehouse Cloud”, a presentation on the agility that Oracle… Author: Mark Rittman Published: 2018-11-14T22:20:36+01:00 Modified: 2018-11-14T22:20:36+01:00 --- One of the sessions we delivered at Oracle Openworld 2018 was “Data Warehouse Like a Tech Startup with Oracle Autonomous Data Warehouse Cloud”, a presentation on the agility that Oracle Autonomous Data Warehouse Cloud (ADWC) can provide for teams within big, traditional businesses that would otherwise have to wait months to provision a physical Oracle data warehousing database. Moreover, ADWC instances can scale-up and scale-down as demand grows over time, and hits peaks such as Black Friday that are only short-term but in the past would have required companies to pay for and provision that level capacity all year round even though it sat idle most of the time. **[Data Warehouse Like a Tech Startup with Oracle Autonomous Data Warehouse](https://www.slideshare.net/markrittman/data-warehouse-like-a-tech-startup-with-oracle-autonomous-data-warehouse-120369905 "Data Warehouse Like a Tech Startup with Oracle Autonomous Data Warehouse")** from **[MJR Analytics](https://www.slideshare.net/markrittman)** The story behind the presentation was also written-up as a [profile article in Forbes magazine](https://www.forbes.com/sites/oracle/2018/10/04/how-a-big-business-can-use-autonomous-it-to-move-like-a-startup/#2e8b685c1752) by Oracle’s Jeff Erickson, with a great description of how MJR Analytics works on client projects. But don’t just take our word for it, or even Forbes Magazine – our objective is for every client engagement to be referenceable where client confidentially allows, and you can check-out our first two testimonials from [Colourpop](https://www.colourpop.com/) and from [Florence](https://www.florenceapp.co.uk/) on our new [Customers](https://www.mjr-analytics.com/customers) page … with a couple more to be added in the next week or so! --- # Event-Level Digital Analytics using Google Analytics, Fivetran, BigQuery and Looker Canonical: https://rittmananalytics.com/blog/2018/11/12/2018-11-12-event-level-digital-analytics-using-google-analytics-fivetran-bigquery-andnbsplooker URL: https://rittmananalytics.com/blog/2018/11/12/2018-11-12-event-level-digital-analytics-using-google-analytics-fivetran-bigquery-andnbsplooker Description: A few weeks ago I posted a blog on using Mixpanel, Looker and Google BigQuery to analyze listener data from the Drill to Detail website, with Mixpanel tracking individual episode… Author: Mark Rittman Published: 2018-11-12T23:16:04+00:00 Modified: 2018-11-12T23:16:04+00:00 --- A few weeks ago I posted a blog on using [Mixpanel, Looker and Google BigQuery to analyze listener data from the Drill to Detail website](https://www.mjr-analytics.com/blog/2018/10/28/nlnhb0kr2emevhbm2qee7jcjot1qtx), with Mixpanel tracking individual episode play events by visitors to the site and Fivetran replicating that event data over to BigQuery for analysis using Looker. Mixpanel is great but like most digital businesses using [Google Analytics](https://marketingplatform.google.com/about/analytics/) (GA) to analyze and measure activity on their website, we actually use [Google Tag Manager](https://marketingplatform.google.com/about/tag-manager/) and GA [events](https://developers.google.com/analytics/devguides/collection/analyticsjs/events) to track visitor activity on the [mjr-analytics.com](https://mjr-analytics.com/) website. Whilst a full site license for Google Analytics 360 is way beyond our budget right now, the free version of GA together with GTM provides us with an unsampled, event-level stream of visitor events and together with goals and simple A/B testing tools a way to start optimizing the content on our site.  Couple all of this with [Fivetran](https://www.mjr-analytics.com/home-index/#dw-services) and [Looker](https://www.mjr-analytics.com/home-index/#looker-services) and we can start to do some interesting analytics on all this data to not only count sessions, visitors and pages viewed but also start looking at visitor retention and the routes those visitors take through the various features on our website. Events in Google Analytics are a way of recording interactions visitors to your site make with items on a page such as selecting an item on a navigation menu, clicking on a download link or seeing a special offer or product recommendation. On the [mjr-analytics.com](https://mjr-analytics.com/) website we’ve setup events, for example, to record how far down the page a visitor scrolls so we can see how many people actually scroll past the [hero image](https://en.wikipedia.org/wiki/Hero_image) on our front page and look at the content on Looker services, Oracle services and so on. Another event records clicks on the specific hero image a visitor clicks on with another recording what navigation menu item was clicked on, in each case storing the image name or menu item clicked on as the event label. Whilst you can record events being triggered by [adding custom Javascript](https://developers.google.com/analytics/devguides/collection/analyticsjs/events) to your website’s HTML page definitions, we use GTM’s graphical point-and-click interface to set-up our events and the visitor actions that trigger them. Enabling GTM’s debug mode for our site shows us which events then trigger when visiting our site, with the screenshot below showing a typical visitor journey through our homepage and the events that are triggered as they scroll-through and interact with various items on the page. Data on events being triggered along with event label and event action values are stored in a Javascript [data layer](https://developers.google.com/tag-manager/devguide) that can also receive values sent over, for example, by an eCommerce application so that clicks on a product catalog item can record the price on offer as well as what product was clicked on.  Looking at the Data Layer view in Google Tag Manager you can see the event entries recorded for this visitor to the site page, but note also how nothing personally identifiable is recorded except for a unique client ID set at the device/browser level that can then be used to analyze new vs. returning visitors and track visitor retention and engagement over time. As we then did with the Mixpanel example and the Drill to Detail website, we then use [Fivetran](https://www.mjr-analytics.com/home-index/#dw-services) to replicate all of the data layer events over to [Google BigQuery](https://www.mjr-analytics.com/home-index/#dw-services), storing this granular visitor behavioural data in a BigQuery table and dataset. As you can see from the screenshot below we use Fivetran extensively within MJR Analytics; replicating invoices and all our other accounting data from Xero into BigQuery along with timesheet data from Harvest, CRM data from Hubspot, commits and PRs from Github, tasks outstanding and completed from Asana and search rankings and keywords from Google Search Console all into a Google BigQuery project whose datasets we then combine and analyze together using Looker. For the event data coming across from GA we use Fivetran to replicate the relevant event tracking dimensions and metrics into BigQuery every five minutes, giving us near real-time numbers on what’s popular and what’s engaging on our company website. As the event data from GA arrives as table rows of individual events being triggered, I then [sessionize those individual events into rows of visitor sessions](https://discourse.looker.com/t/join-2017-deep-dive-event-sessionization/5835), using BigQuery SQL to pivot the incoming data and create individual columns for the first event, second event, first page view and first product item clicked on, for example, within an individual visitor session. The actual SQL used for the view is too long really to include here but to get the idea, here’s a condensed version with repeating parts removed for clarity. SELECTglobal\_session\_id,visitor\_session\_id,visitor\_id,visitor\_session\_start\_date\_time,MAX(visitor\_session\_length\_mins) AS visitor\_session\_length\_mins,visitor\_first\_event\_week\_cohort,city,session\_events\_count,…event\_1\_category,event\_1\_label,…page\_view\_1,page\_view\_2,…FROM (SELECTe.global\_session\_id,e.visitor\_session\_id,e.visitor\_id,e.visitor\_session\_start\_date\_time,e.visitor\_session\_length\_mins,e.visitor\_first\_event\_week\_cohort,e.city,MAX(e.visitor\_session\_event\_seq\_num) OVER (PARTITION BY e.global\_session\_id) AS session\_events\_count,(SELECTCOUNT(\*)FROM\`aerial-vehicle-148023.dw\_staging.ga\_events\`WHEREglobal\_session\_id = e.global\_session\_idAND event\_category = ‘Page View’) AS page\_view\_count,SELECTCOUNT(\*)FROM\`aerial-vehicle-148023.dw\_staging.ga\_events\`WHEREglobal\_session\_id = e.global\_session\_idAND event\_category = ‘Nav Bar Clicks’) AS nav\_bar\_click\_count,…,e1.event\_label AS event\_1\_label,e2.event\_category AS event\_2\_category,e2.event\_label AS event\_2\_label,…pv.event\_label AS page\_view\_1,pv2.event\_label AS page\_view\_2,…,igc.event\_label AS index\_gallery\_first\_clickFROM\`aerial-vehicle-148023.dw\_staging.ga\_events\` eLEFT OUTER JOIN (SELECTglobal\_session\_id,event\_label,ROW\_NUMBER() OVER (PARTITION BY global\_session\_id ORDER BY visitor\_session\_event\_seq\_num) page\_view\_seq\_numFROM\`aerial-vehicle-148023.dw\_staging.ga\_events\`WHEREevent\_category = ‘Page View’ORDER BY3) pvONe.global\_session\_id = pv.global\_session\_idAND pv.page\_view\_seq\_num = 1LEFT OUTER JOIN (SELECTglobal\_session\_id,event\_label,ROW\_NUMBER() OVER (PARTITION BY global\_session\_id ORDER BY visitor\_session\_event\_seq\_num) page\_view\_seq\_numFROM\`aerial-vehicle-148023.dw\_staging.ga\_events\`WHEREevent\_category = ‘Page View’ORDER BY3) pv2ONe.global\_session\_id = pv2.global\_session\_idAND pv2.page\_view\_seq\_num = 2…JOIN\`aerial-vehicle-148023.dw\_staging.ga\_events\` e1ONe.global\_session\_id = e1.global\_session\_idAND e1.visitor\_event\_seq\_num = 1LEFT OUTER JOIN\`aerial-vehicle-148023.dw\_staging.ga\_events\` e2ONe.global\_session\_id = e2.global\_session\_idAND e2.visitor\_event\_seq\_num = 2…GROUP BY1,2,…,46ORDER BY1 Creating a view or materialized table from this BigQuery SQL then gives me one record per session with event totals, page view and event paths along with session and client IDs that are perfect then for bringing into Looker for further analysis. Then finally it’s just a case of bringing that table of data into Looker as a LookML view and using one of Looker’s custom visualization blocks to display visitor event paths as a Sankey diagram, for example. Or we can aggregate and analyze counts of events from sessions to see in graphical form which offers, download items and navigation menu items are most popular over a given period. If you’d like to understand more about how MJR Analytics can help you better understand what visitors are looking at and interacting with on your website then drop me an email at [mark.rittman@mjr-analytics.com](mailto:mark.rittman@mjr-analytics.com); or even better, if you’re in Copenhagen on November 22nd and have the evening free, come along to the [free meetup we’re hosting together with our partners CIMA and Looker](http://linksplit.io/MTEyMTA=) and you’ll see me demo this all live, in person and show you how it works. --- # Digital Analytics, BI and Big Data Meetup in Copenhagen 22.11.2018 Canonical: https://rittmananalytics.com/blog/2018/11/11/digital-analytics-bi-and-big-data-meetup-in-copenhagen-22-11-2018 URL: https://rittmananalytics.com/blog/2018/11/11/digital-analytics-bi-and-big-data-meetup-in-copenhagen-22-11-2018 Description: If you’re in Copenhagen on 22nd November 2018 and interested in digital analytics, big data and what’s new in business intelligence then you’ll be interested in a free evening meetup… Author: Mark Rittman Published: 2018-11-11T23:36:06+01:00 Modified: 2018-11-11T23:36:06+01:00 --- If you’re in Copenhagen on 22nd November 2018 and interested in [digital analytics, big data and what’s new in business intelligence](http://linksplit.io/MTEyMTA=) then you’ll be interested in a free evening meetup we’re hosting that evening with our partners CIMA and Looker: [Register Now](http://linksplit.io/MTEyMTA=) **[“Where Digital Analytics is Taking BI and Big Data”](http://linksplit.io/MTEyMTA=)** How usage and new vendors in the digital, eCommerce and SaaS Analytics market are revolutionising the way in which data is analysed, democratising big data and enabling businesses to finally become “data-driven”. - Introduction by Mogens Norgaard, CIMA - Presented by Mark Rittman, CEO, MJR Analytics Meetup hosted by CIMA Technologies, MJR Analytics and Looker **Date**: Thursday 22nd November **Time**: 17.00hrs – 19.00hrs **Location**: Ingeniørforeningen IDA, Kalvebod Brygge 31-33 · 1780 København V Registration is free, and there’ll be talks by myself and Looker with an introduction by none other than Mogens Norgaard. Just don’t let Mogens know it’s my birthday that day too, or at least not until I’ve finished my presentation and demo – hopefully we’ll see some of you in Copenhagen for the meetup, it’ll be good. --- # Digital Analytics, BI and Big Data Meetup in Copenhagen 22.11.2018 Canonical: https://rittmananalytics.com/blog/2018/11/11/2018-11-11-2018-11-11-digital-analytics-bi-and-big-data-meetup-in-copenhagen-22111967 URL: https://rittmananalytics.com/blog/2018/11/11/2018-11-11-2018-11-11-digital-analytics-bi-and-big-data-meetup-in-copenhagen-22111967 Description: If you’re in Copenhagen on 22nd November 2018 and interested in digital analytics, big data and what’s new in business intelligence then you’ll be interested in a free evening meetup… Author: Mark Rittman Published: 2018-11-11T22:36:06+00:00 Modified: 2018-11-11T22:36:06+00:00 --- If you’re in Copenhagen on 22nd November 2018 and interested in [digital analytics, big data and what’s new in business intelligence](http://linksplit.io/MTEyMTA=) then you’ll be interested in a free evening meetup we’re hosting that evening with our partners CIMA and Looker: [Register Now](http://linksplit.io/MTEyMTA=) **[“Where Digital Analytics is Taking BI and Big Data”](http://linksplit.io/MTEyMTA=)** How usage and new vendors in the digital, eCommerce and SaaS Analytics market are revolutionising the way in which data is analysed, democratising big data and enabling businesses to finally become “data-driven”. - Introduction by Mogens Norgaard, CIMA - Presented by Mark Rittman, CEO, MJR Analytics Meetup hosted by CIMA Technologies, MJR Analytics and Looker **Date**: Thursday 22nd November **Time**: 17.00hrs – 19.00hrs **Location**: Ingeniørforeningen IDA, Kalvebod Brygge 31-33 · 1780 København V Registration is free, and there’ll be talks by myself and Looker with an introduction by none other than Mogens Norgaard. Just don’t let Mogens know it’s my birthday that day too, or at least not until I’ve finished my presentation and demo – hopefully we’ll see some of you in Copenhagen for the meetup, it’ll be good. --- # Media Analytics for the Drill to Detail Podcast using Looker, Fivetran and Mixpanel Canonical: https://rittmananalytics.com/blog/2018/10/28/2018-10-28-nlnhb0kr2emevhbm2qee7jcjot1qtx URL: https://rittmananalytics.com/blog/2018/10/28/2018-10-28-nlnhb0kr2emevhbm2qee7jcjot1qtx Description: Most readers of the MJR Analytics blog will no doubt be aware of the Drill to Detail Podcast on the big data analytics industry that we present and host, with… Author: Mark Rittman Published: 2018-10-28T14:11:29+00:00 Modified: 2018-10-28T14:11:29+00:00 --- Most readers of the MJR Analytics blog will no doubt be aware of the [Drill to Detail Podcast](https://www.mjr-analytics.com/podcast) on the big data analytics industry that we present and host, with the [most recent episode](https://www.mjr-analytics.com/podcast) covering the Looker JOIN 2018 event in San Francisco with Tristan Handy and Stewart Bryson as special guests. Things have been a bit busy over the past month with the launch of MJR Analytics and then the Looker and Oracle conferences but we try and upload a new episode every few weeks, with the next episode — fingers crossed — having a rather special guest if we can get our schedules aligned. Listeners to the show typically either use services such as [Apple’s iTunes Podcast Directory](https://itunes.apple.com/gb/podcast/drill-to-detail/id1141126960?mt=2) to download episodes and play them typically on their iOS mobile devices or they can visit the show’s website at [https://www.drilltodetail.com](https://www.drilltodetail.com/) where they can download or play episodes through an embedded audio player and click through the links in the show notes that accompany each episode. Apple provides statistics at episode level to podcast owners via their new Podcast Analytics service but you’re limited to just the charts and fixed ways that the app allows you to analyze and explore your dataset. If you can find a way to scrape or download this data and load it into a database service such as Google BigQuery though, then you could instead use a tool such as Looker to analyze and visualize your listener data in more interesting ways that are also very applicable to customer and product analytics for any digital or eCommerce site, as I’ll explain as we go through the examples. To start, the Looker visualization below shows for each episode of the show how many in-total of that episode were listened to by users of Apple’s Podcasts Directory service on their iPhones, iPads and Mac OS devices, with the single most listened-to episode of all time within a given month being our [50th Episode Special that featured Stewart Bryson and Alex Gorbachev](https://www.mjr-analytics.com/podcast/2018/02/27/2018-2-27-drill-to-detail-ep50-agile-bi-karl-marx-and-our-man-from-moscow-with-special-guests-stewart-bryson-and-alex-gorbachev) as our two special guests. Apple also provide data on the how much of each episode was completed by the listener, so you can get a sense of how long episodes should last and how engaged your audience was with individual episodes. In the Looker visualization below you can see that typically our listeners get through around half to two-thirds of each episode but were particularly engaged with Wes McKinney episode on Apache Arrow, Python Pandas and his blog post comparing [Arrow with ORC and Parquet](http://wesmckinney.com/blog/arrow-columnar-abadi/) that was generating a lot of discussion in the open-source and big data community at the time. So far the way I’ve analysed downloads and listener numbers for the podcast is not too dissimilar to how businesses used to only measures sales per month but didn’t really think about how many of their customers and visitors coming onto their site each month were new or in-fact returning customers looking to buy more.  These days knowing how many of your customers are new or returning is the first thing you’d want to understand as an eCommerce manager because it’s cheaper to sell to an existing customer than to go out and recruit a new one, and one of your main priorities is to try and turn those first-time buyers into loyal, repeat purchasers who then become your most valuable customers. My focus on episodes rather than listeners was in-reality more due to Apple not providing episode download and listening data at the granularity of an individual user or device ID, but some time ago I also set up Mixpanel, a Javascript event-tracking service similar to Heap and Google Analytics’s custom events, to track when visitors played episodes of the show on the site through their browser as shown in the screenshots below. Data is still only logged at unique device level as there’s no registration facility on the site or other means to tie downloads to particular users, but now we have access to additional details of each download such as where the user was located and what type of browser and OS they were using. Mixpanel stores the events it tracks in a database service it hosts, but you can use a managed data replication service such as the one from our partners at Fivetran, shown in the screenshots below, to copy that data into a warehousing platform such as BigQuery as I’ve done and shown below. Going back to Looker I now create two new LookML views against the single table of Mixpanel tracked events that Fivetran replicated into BigQuery for me, the first of which is keyed on the unique distinct\_id columns and contains dimensions and measures for the listener’s device used for accessing the site and playing-back episodes of the show. view: unique\_devices {derived\_table: {sql: SELECTdistinct\_id,device,mp\_country\_code,os, region, screen\_height, screen\_widthFROM\`aerial-vehicle-148023.mixpanel.event\`WHEREname IN (‘Clicked Nav Link’,’Podcast Episode Played’)group by 1,2,3,4,5,6,7 ;;}dimension: distinct\_id {primary\_key: yeslabel: “Device ID”}dimension: device {sql: case when ${TABLE}.os in (‘Windows’,’Mac OS X’,’Linux’) then ‘Desktop’when ${TABLE}.os = ‘Chrome OS’ then ‘Chromebook’else device end;;}dimension: os {}dimension: region {}dimension: screen\_height {}dimension: screen\_width {}dimension: mp\_country\_code {label: “Country”}measure: count\_unique {type: count\_distinctsql: ${TABLE}.distinct\_id ;;}measure: count {type: count}} The second view is keyed on event\_id and contains the dimensions and measures used for analyzing and counting playback events by those listeners’ devices. — page events view definitionview: page\_events {derived\_table: {sql: SELECTREPLACE(JSON\_EXTRACT(e.properties,’$.Episode’),’”‘,”) AS episode,REPLACE(JSON\_EXTRACT(e.properties,’$.url’),’”‘,”) AS nav\_url,episode\_number,distinct\_id,current\_url,event\_id,initial\_referrer,initial\_referring\_domain,name as event\_type,referrer,referring\_domain,search\_engine,timeFROM\`aerial-vehicle-148023.mixpanel.event\` eleft outer join \`aerial-vehicle-148023.dw\_staging.episode\_list\` lon REPLACE(JSON\_EXTRACT(e.properties,’$.Episode’),’”‘,”) = l.episodeWHEREe.name IN (‘Clicked Nav Link’,’Podcast Episode Played’) ;;}dimension\_group: event {type: timetimeframes: \[date,week,month,hour\_of\_day,day\_of\_week,day\_of\_month\]sql: ${TABLE}.time ;;}dimension: event\_type {}dimension: event\_id {primary\_key: yeshidden: yes}dimension: distinct\_id {hidden: yes}dimension: episode {sql: replace(${TABLE}.episode,’Drill to Detail’,”) ;;}dimension: episode\_number {sql: lpad(cast(${TABLE}.episode\_number as string),2,’0′) ;;}dimension: nav\_url {}dimension: referring\_domain {}dimension: search\_engine {}measure: event\_count {type: count}measure: unique\_device\_count {type: count\_distinctsql: ${TABLE}.distinct\_id ;;}measure: unique\_devices {type: count\_distinctsql: ${TABLE}.distinct\_id ;;}measure: episode\_num {type: averagesql: ${TABLE}.episode\_number;;}} Then, a new explore is defined within the LookML project to join these two views and make them available for analysis in the Looker Explore page. — explore definitionexplore: page\_events {join: unique\_devices {relationship: one\_to\_manysql\_on: ${page\_events.distinct\_id} = ${unique\_devices.distinct\_id} ;;}} Creating a look similar to the one a moment ago but using episode play events recorded by Mixpanel as the measure, I can see that whilst most episodes had the majority of downloads in the month they were published, [the episode featuring Jonathan Palmer from King Games](https://www.mjr-analytics.com/podcast/2018/07/03/2018-7-3-drill-to-detail-ep58-candy-crush-saga-analytics-and-mobile-gaming-with-special-guest-jonathan-palmer) on how they used Looker to design and optimize games such as Candy Crush Saga had downloads well after the initial month it became available, telling me that content on this type of topic has perhaps a longer “shelf-life” than others we’ve covered. If I use the “merge results” feature introduced with Looker 5 to combine results from two or more queries, I can combine the download numbers per episode per month per unique device ID from both Apple and from Mixpanel’s tracking of plays on the website to give me a combined listener count for each episode, and you can see from the screenshot below that the Episode 50 Special is still the most downloaded episode and the later one with Jonathan Palmer does seem to still be listened to for a lot longer than most other episodes. Except it’s not. What the chart above actually shows is that Episode 50 was the most downloaded in any one month, but if you take each episode and chart total downloads overall it’s actually the Christmas and New Year Special featuring Christian Berg that overall had more unique device downloads. Putting the above two merged data looks into a dashboard along with two others showing the split by iTunes and website downloads over time and those same numbers expressed as share of all downloads over time, you can get a sense of for how long each episode was downloaded after first publication, the split for each episode by download platform and how listener numbers for each method of download have changed over time. You can see next chart how downloads via iTunes have a close correlation with new episodes being published with listener numbers dropping over the summer when the show took a break; in contrast, downloads directly from the website held-up even over the summer break when no new episodes were published. Taking just the device-level data for website downloads coming from Mixpanel and creating a dashboard from the referrer and device details attributes it confirms as I’d have expected that most of those downloads come from listeners using Windows desktop PCs or Android phones, given iTunes’ focus on MacOS and iOS devices, but I can also see that most plays coming from search traffic come from Google and occasionally from Bing, and that Christian Berg’s listeners were unusually-spread amongst lots of different countries, not just the usual UK, USA, Germany and so on. But what none of these dashboards tell me is how good I am at retaining listeners after they’ve downloaded their first ever episode; although I suspect many listeners come for one particular episode I’d be interested to know how many devices we see each month are returning or new and we can of course extend the scope of website activity to include clicks on navigation links as another indicator of engagement and retention. To do this we need to know, for every visitor unique device ID, what sequence number in the total set of episode plays for that device this particular play represents; if the sequence number for a given play event was 1 then we’d classify the visitor as new, whereas if the sequence number was greater than 1 then we’d classify the visitor as returning. Sequencing these device playback events also then makes it possible to analyze play histories for individual podcast listener devices to understand how many episodes a typical listener plays and whether a particular episode triggers more repeat listenings of the show than others, to understand the frequency of listening and whether returning visitors are coming back to play the same episode or taking the opportunity to discover and play other episodes from the site — exactly the sort of analysis that online retailers do when looking to turn casual browsers and one-off purchasers into loyal, repeat buyers from their site. To do this I create an SQL query in Looker’s SQL Runner utility that returns the episodes played for each visitor distinct\_id and adds a ROW\_NUMBER window function to provide a sequence number for the episode plays sorted in order of time To add this query to my Looker project as a derived table I then select **Get Derived Table LookML** from the menu in the top right-hand side of the page, like this: SELECTdistinct\_id,event\_id,time,episode\_number,row\_number() over (partition by distinct\_id order by time) as play\_order\_seqFROM\`aerial-vehicle-148023.mixpanel.event\` eLEFT OUTER JOIN \`aerial-vehicle-148023.dw\_staging.episode\_list\` lON REPLACE(JSON\_EXTRACT(e.properties,’$.Episode’),’”‘,”) = l.episodeWHEREe.name = ‘Podcast Episode Played’ and then choose the option to add it to your project when the **Get Derived Table** LookML dialog is shown. Then, after setting all of the derived table columns as hidden and the event\_id column as the primary key for this view, I add one new dimension to the view that returns “new” or “returning” depending on whether the play order sequence number for the play event is 1 or any other number and also set the view\_label for this view to the same name as the main page events view, so that the play order sequence number appears alongside the other play event fields in the main explore view. view: website\_play\_sequence {view\_label: “Page Events”derived\_table: {sql: SELECTdistinct\_id,event\_id,time,row\_number() over (partition by distinct\_id order by time) as play\_order\_seqFROM\`aerial-vehicle-148023.mixpanel.event\`WHEREname = ‘Podcast Episode Played’;;}dimension: distinct\_id {hidden: yestype: stringsql: ${TABLE}.distinct\_id ;;}dimension: event\_id {hidden: yesprimary\_key: yessql: ${TABLE}.event\_id ;;}dimension\_group: time {hidden: yestype: timesql: ${TABLE}.time ;;}dimension: play\_order\_seq {type: numbersql: ${TABLE}.play\_order\_seq ;;}dimension: new\_or\_returning {type: stringsql: case when ${TABLE}.play\_order\_seq = 1 then ‘new’ else ‘returning’ end ;;}} I then join this new derived table to the page events view in the existing model explore on the event\_id column common to both of them, like this: explore: page\_events {join: unique\_devices {relationship: one\_to\_manysql\_on: ${page\_events.distinct\_id} = ${unique\_devices.distinct\_id} ;;}join: website\_play\_sequence {relationship: one\_to\_manysql\_on: ${page\_events.event\_id} = ${website\_play\_sequence.event\_id} ;;} Now I can create a look that shows episode play events over time broken down by new or returning device, and I can see that about two-thirds of listens each month come from returning devices. If we pivot on the episode play sequence number and then turn the episode numbers themselves into a measure, we can then show for each individual device the sequence of episodes they’ve downloaded. So far we’ve looked at devices playing podcast episodes on the site as one, homogenous group, working on the assumption that listeners visiting the site for the first time now are no different from ones who found us a year or more ago, and the site itself hasn’t changed over that period of time.  But in reality we may have improved the usability of the site or made it easier for visitors to find and discover other episodes, and so I’m keen to understand whether the groups, or “cohorts” of users who joined us in more recent months are engaged for longer and churn at a slower rate than visitors who found the site before those changes were made. I can return the month that each visitor’s device first played a podcast episode on the site using another SQL query, and whilst I’m pulling this together I might as well calculate how long each device has been accessing episodes on the site along with totals for episode play events and distinct episodes listened to for each visitor device. SELECT distinct\_id,date\_trunc(date(min(time)),MONTH) as first\_visit\_at,timestamp\_diff(max(time),min(time),day)+1 as listener\_days,count(\*) as total\_plays,count(distinct JSON\_EXTRACT(properties,’$.Episode’)) as total\_episodesFROM mixpanel.eventWHERE name = ‘Podcast Episode Played’GROUP BY 1 I then use the feature in SQL Runner that generates a derived table view definition from an SQL query and then fine-tune the dimension definitions and convert the listener\_days, total\_plays and total\_episode columns into a set of appropriate measures. view: visitor\_cohort {view\_label: “Unique Devices”derived\_table: {sql: SELECT distinct\_id,date\_trunc(date(min(time)),MONTH) as first\_visit\_at,timestamp\_diff(max(time),min(time),day)+1 as listener\_days,count(\*) as total\_plays,count(distinct JSON\_EXTRACT(properties,’$.Episode’)) as total\_episodesFROM mixpanel.eventWHERE name = ‘Podcast Episode Played’GROUP BY 1;;}dimension: distinct\_id {primary\_key: yeshidden: yestype: stringsql: ${TABLE}.distinct\_id ;;}dimension: first\_visit\_at {type: datelabel: “Listener Cohort”sql: ${TABLE}.first\_visit\_at ;;}measure: listener\_days {type: averagesql: ${TABLE}.listener\_days ;;}measure: avg\_episode\_plays {type: averagesql: ${TABLE}.total\_plays ;;}measure: total\_episode\_plays {type: sumsql: ${TABLE}.total\_plays ;;}measure: avg\_episodes {type: averagesql: ${TABLE}.total\_episodes ;;}} Then I join this second derived table into the model explore, this time joining to the unique\_devices view on their common distinct\_id column as the additional columns I’m now adding in apply to visitor devices, not episode play events as in the previous example. explore: page\_events {join: unique\_devices {relationship: one\_to\_manysql\_on: ${page\_events.distinct\_id} = ${unique\_devices.distinct\_id} ;;}join: website\_play\_sequence {relationship: one\_to\_manysql\_on: ${page\_events.event\_id} = ${website\_play\_sequence.event\_id} ;;}join: visitor\_cohort {relationship: one\_to\_onesql\_on: ${unique\_devices.distinct\_id} = ${visitor\_cohort.distinct\_id} ;;}} I can now create looks such as the one below, where I chart the size of each cohort and overlay the average number of episode each visitor device goes on to download to give me a combined view of how many new listeners we recruit each month and whether each group over time listens to more, or less episodes than the previous ones. If I now add two more window function calculations to the SQL query used by the derived table that provides the sequence number for each episode play by a device, like this: SELECTdistinct\_id,event\_id,time,row\_number() over (partition by distinct\_id order by time) as play\_order\_seq,date\_diff(date(time),date(first\_value(time) over (partition by distinct\_id order by time)),MONTH) as months\_since\_first\_play\_at,date\_diff(date(time),date(lag(time,1) over (partition by distinct\_id order by time)),DAY) as days\_since\_last\_playFROM\`aerial-vehicle-148023.mixpanel.event\`WHEREname = ‘Podcast Episode Played’ I can now create a heat-map for each cohort showing how their engagement maintains or falls-off for the six months after their first play of a podcast episode on the site. If I switch the two dimensions around and change the visualization type to a line chart using a logarithmic scale, you can now see how engagement falls-off (or “decays”) for each of my listener cohorts over the six months since they first play an episode on the site. Or, finally, I could take the Days Since Last Episode Play measure and use it to show how time between episode plays increases for each cohort, with big increases in time elapsed being a strong indicator of that cohort losing interest in the podcast and “churning”. So whilst we’ve been analyzing listeners to a podcast and in-detail, only a small and perhaps skewed subset of overall listeners, the techniques we’ve used including cohorting visitors based on when they first download or purchase and then tracking their engagement and overall lifetime value are ones we’ve used recently when working on Looker projects for [clients such as Colourpop in the US and Florence back home in the UK](https://www.mjr-analytics.com/customers).  Drop us an email at [info@mjr-analytics.com](mailto:info@mjr-analytics.com) or contact us on +44 7866 568246 if you’re looking to do something similar with your customer or product data, and in the meantime you can check out past episodes of the Drill to Detail podcast a the new home for the show, co-located with this blog on the MJR Analytics site at [https://www.mjr-analytics.com/podcast](https://www.mjr-analytics.com/podcast). --- # MJR Analytics Presenting at Oracle Openworld 2018, San Francisco Canonical: https://rittmananalytics.com/blog/2018/10/22/mjr-analytics-presenting-at-oracle-openworld-2018-san-francisco URL: https://rittmananalytics.com/blog/2018/10/22/mjr-analytics-presenting-at-oracle-openworld-2018-san-francisco Description: Mark Rittman from MJR Analytics will be presenting at this week’s Oracle Openworld 2018 event in San Francisco, both sessions on Monday, 22nd October 2018. BI Developer to Data Engineer… Author: Mark Rittman Published: 2018-10-22T02:00:42+01:00 Modified: 2018-10-22T02:00:42+01:00 --- Mark Rittman from MJR Analytics will be presenting at this week’s [Oracle Openworld 2018](https://www.oracle.com/openworld/index.html) event in San Francisco, both sessions on Monday, 22nd October 2018. **BI Developer to Data Engineer with Oracle Analytics Cloud, Data Lake \[PRO3189\]** Monday, Oct 22, 11:30 AM – 12:15 PM | Marriott Marquis (Yerba Buena Level) – Nob Hill C/D > “In this session look at the role of a data engineer in designing, provisioning, and enabling an Oracle Cloud data lake using Oracle Analytics Cloud, Data Lake. Attendees learn how to use data flow and data pipeline authoring tools and how machine learning and AI can be applied to this task, as well as how to connect to database and SaaS sources along with sources of external data via Oracle Data as a Service. Discover how traditional Oracle Analytics developers can transition their skills into this role and start working as a data engineer on Oracle Public Cloud data lake projects.” **Data Warehouse Like a Tech Startup with Oracle Autonomous Data Warehouse \[BUS3194\]** Monday, Oct 22, 04:45 PM – 05:30 PM | Moscone West – Room 3006 > “Tech startups can’t afford DBAs, and they don’t have time to provision servers and scale them up and down or deal with patches or downtime. They’ve never heard of indexes and they need data loaded and ready for analysis in days, not months. In this session learn how Oracle Database developers can build data warehouses as a hip startup data engineer would—but using a proper database built on Oracle technology. Oracle Data Visualization Desktop provides analytics and data exploration with techniques explained in this session. Hear real-world development experiences from working on data and analytics projects at a tech startup in the UK.” Mark will be at Openworld and around San Francisco until the end of the event, so if you’d like to have a chat about these talks or anything to do with Oracle Analytics then get in touch at [mark.rittman@mjr-analytics.com](mailto:mark.rittman@mjr-analytics.com) or call us on +44 7866 568246. --- # Introducing MJR Analytics … and How Two Years Go So Fast When You’re Learning Something New Canonical: https://rittmananalytics.com/blog/2018/09/27/introducing-mjr-analytics-and-how-two-years-go-so-fast-when-youre-learning-something-new URL: https://rittmananalytics.com/blog/2018/09/27/introducing-mjr-analytics-and-how-two-years-go-so-fast-when-youre-learning-something-new Description: Today I’m excited to be launching MJR Analytics, a new consulting company focusing on modern, cloud analytics projects using technology from Looker, Qubit, Fivetran, Oracle, Google Cloud Platfand Snowflake and… Author: Mark Rittman Published: 2018-09-27T23:01:11+01:00 Modified: 2026-07-02T00:38:29+01:00 --- ![Introducing MJR Analytics … and How Two Years Go So Fast When You’re Learning Something New](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F56b56-1depqrpwrgm8zvqcissuz9a.png) Today I’m excited to be launching [MJR Analytics](https://mjr-analytics.com/), a new consulting company focusing on modern, cloud analytics projects using technology from Looker, Qubit, Fivetran, Oracle, Google Cloud Platfand Snowflake and development techniques learnt from my time working as an analytics product manager at a startup in London. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F56b56-1depqrpwrgm8zvqcissuz9a.png) Our new website (and no that’s not me sitting in the chair) So what have I been up to in the two years since I left my old consulting company, and how has that experience and the way I’ve been working with analytics technologies over that time inspired me to start another one? Come over to our new company website and [read the fully story](https://www.mjr-analytics.com/blog/2018/09/17/introducing-mjr-analytics-and-how-two-years-go-so-fast-when-youre-learning-something-new). --- # Introducing MJR Analytics … and How Two Years Go So Fast When You’re Learning Something New Canonical: https://rittmananalytics.com/blog/2018/09/17/2018-09-17-2018-09-17-introducing-mjr-analytics-and-how-two-years-go-so-fast-when-youre-learning-something-new URL: https://rittmananalytics.com/blog/2018/09/17/2018-09-17-2018-09-17-introducing-mjr-analytics-and-how-two-years-go-so-fast-when-youre-learning-something-new Description: Today I’m excited to be launching MJR Analytics, a new consulting company focusing on modern, cloud analytics projects using technology from Looker, Qubit, Fivetran, Oracle and Snowflake and development techniques… Author: Mark Rittman Published: 2018-09-17T09:17:06+00:00 Modified: 2018-09-17T09:17:06+00:00 --- Today I’m excited to be launching [MJR Analytics](https://mjr-analytics.com/), a new consulting company focusing on modern, cloud analytics projects using technology from Looker, Qubit, Fivetran, Oracle and Snowflake and development techniques learnt from my time working as an analytics product manager at a startup in London. Our new website (and no that’s not me sitting in the chair) So what have I been up to in the two years since I left my old consulting company, and how has that experience and the way I’ve been working with analytics technologies over that time inspired me to start another one? Two years ago I announced on Twitter that I’d left the company I’d co-founded back in 2007 and intended to now take on a new challenge, and then spent the rest of the week at Oracle Open World cycling over the Golden Gate Bridge and coming back on the ferry and trying to decide what that challenge might actually be. > Last Friday was my final day at Rittman Mead, and I wish the team well for the future. Time for a new challenge, and at [#oow16](https://twitter.com/hashtag/oow16?src=hash&ref_src=twsrc%5Etfw) all this week > > — Mark Rittman (@markrittman) [September 19, 2016](https://twitter.com/markrittman/status/777915459093929984?ref_src=twsrc%5Etfw) [https://platform.twitter.com/widgets.js](https://platform.twitter.com/widgets.js) For most of my time in the IT industry I’d been working on projects implementing products from vendors such as Oracle and I’d always been interested in how these products came to market, how software vendors came up with a strategy and roadmap for those products how the team behind those products worked with the engineers who built them. I’d also become increasingly interested in the startup world and towards the end of time time at Rittman Mead had taken-on an informal role advising [Gluent](https://gluent.com/), Tanel Poder and Paul Bridger’s product startup who were building software that enabled big enterprise customers to offload their data warehousing workloads from expensive proprietary databases onto to cheap, flexible storage and processing running on Hadoop clusters. What appealed to me about working more formally with Gluent was the opportunity it gave me to work with two smart founders and an even smarter development team developing a product built entirely on big data technology I’d until then only scratched the surface with on consulting gigs. The product marketing role I took on was all about establishing what market that product was primarily intended for, how we went about positioning the product to appeal to that market and how we then brought that product to market. Understanding these four things are crucial if you’re going to actually get customers to buy your startup’s product: - who is the buyer - what problem does your product solve - what is the value solving that problem, and - why you’re the first product to solve it for them otherwise you’ll spend your time building a solution to a problem that nobody actually has, and that’s the reason the majority of tech startups end-up failing. Solving a problem for a market that’s willing to pay you money to solve is called “product/market fit” and if your product has it, and you’ve built your business such that it scales linearly as more customers discover your product, then you’re going to make a lot more money than a consultancy constrained by how many hours in the week your consultants can work and the upper limit on how much you can charge for a single person’s time. I also learnt the distinction between product marketing, product management and product development in my time at Gluent. Going back to my time as a consultant attending product roadmap sessions at conferences I never quite knew which parts of the product team those speakers came from, but in summary: - *Product Marketing* is about taking a product that’s typically already built and then deciding the product’s positioning and messaging, then launching the product and ensuring the sales team, sales engineering and customers understand how it works and what it does; as such, this is a marketing role with a bit of technical evangelism thrown in - *ProductDevelopment* is the actual building of the product you’re looking to sell, and requires an engineering skillset together with the inspiration that typically came up with the product idea in the first place along and an entrepreneurial side that made you want to build a company around it - *Product Management* is more of a customer-facing role and is about understanding what your customers want and what their problems and use-cases are, and then creating a strategy, roadmap and feature definition for a product that will meet those needs Despite my undoubted product marketing skills based around PowerPoint and internet memes: In product marketing, it’s never too soon to put a Santa hat on a photo of the founder in the end it I realised that it was product management that interested me the most and, after a couple of meetings with an old friend who used to run product management at Oracle for their business analytics product line and who had recently moved to London and now lead the product team team at Qubit, a technology startup created by four ex-Googlers building marketing technology products based-around Google’s big data and cloud technology, I joined their team later in 2016 as product manager responsible for the analytics features on their platform. I spoke about the Qubit and the partnership we established with Looker back in May last year at a presentation at Looker’s JOIN 2017 conference in San Francisco and the slide deck below from that event goes into the background to the product and the problem it solves, helping customers using Qubit’s personalization platform make more effective use of the data we collected for them. **[Petabytes to Personalization – Data Analytics with Qubit and Looker](https://www.slideshare.net/markrittman/petabytes-to-personalization-data-analytics-with-qubit-and-looker)** from **[MJR Analytics](https://www.slideshare.net/markrittman)** The product and data engineering teams at Qubit did an excellent job bringing together the features for this product and in hindsight, the bits I was most proud of included: - The business metadata layer we created on Google BigQuery and Google Cloud Platform to translate an event-level normalized many-to-many data model designed for fast data ingestion into a denormalized, dimensional data model designed for easy use with BI and ETL tools - Additional integration we created for the Looker BI tool including a set of industry vertical-specific Looker models and dashboards we then made available on the [Looker Block Directory](https://looker.com/platform/blocks/source/personalization-analytics-by-qubit) and in a [Github public repo](https://github.com/QubitProducts/looker-source-block/blob/master/README.md) Screenshot from Personalization Analytics Block for Looker by Qubit - The multi-tenant data warehouse and hosted Looker instance we then put together to enable customers without their own Looker instance to make use of their data in Google BigQuery, doing so in a way that supported per-tenant extensions and customizations by the customer or their implementation partner. Technical Architecture for Live Tap as presented at Looker JOIN 2017 What I’ll take-away from my time at Qubit though was the incredible amount that I learnt about product management, product engineering, how to build and run a successful startup and team who are still highly-motivated seven years in and the introduction it gave me to the analytics and data-led world of digital marketing, eCommerce and modern data analytics platforms. Consulting is a popular route into product management and the experience I brought to the role in areas such as business metadata models, analytical techniques and the needs of BI and ETL developers proved invaluable over the eighteen months I worked as part of Qubit’s product and engineering teams, but moving into product management within a young, technology-led startup founded by ex-Googlers and working with some of the smartest and most innovative people I’ve ever met involved learning a whole new set of skills including: - Developing on a new technology platform (Google Cloud Platform) within a new industry (eCommerce and digital marketing) and understanding a whole new set of analytics use-cases and customer roles (A/B testing, stats models and event-based analytics used by analysts and strategists within eCommerce businesses) that I described in a presentation at last year’s UK Oracle User Group Tech Conference in Birmingham: **[Analytics is Taking over the World (Again) – UKOUG Tech’17](https://www.slideshare.net/markrittman/analytics-is-taking-over-the-world-again-ukoug-tech17)** from **[MJR Analytics](https://www.slideshare.net/markrittman)** - Working as part of a team rather than directing that team, and managing -up as well as down, a technique I had to relearn pretty quickly in my first few months in the role - Learning to achieve my goals through influence rather than in the top-down way I’d been used to getting things done leading customer projects, and as CTO and owner of the company that team worked for - Saying no to customers rather than yes as you did as a consultant, as your objective is to build a product that solves the most important customer needs but doesn’t burden you with so many features addressing niche use-cases that you end up with Homer’s car and can’t innovate the product in future releases - How to take a product through its lifecycle from identifying a need that makes sense for your company to meet, through prototyping, alpha and beta releases to successful first launch and then creating a strategy and roadmap to manage that product over its complete lifecycle - How to use a new generation of modern, cloud-native data analytics tools such as Looker together with products such as [FiveTran](https://fivetran.com/), [Google Cloud Platform](https://cloud.google.com/), [Qubit](https://qubit.com/), [Snowflake DB](https://www.snowflake.com/) and [Snowplow Analytics](https://snowplowanalytics.com/) that were increasingly also being adopted by the FinTech, MarTech and B2C startups clustering in London and other European/North American tech hubs I learnt so much from my colleagues at Qubit about products, engineering and building a successful and motivated team that put up with my jokes and built the most technologically-advanced marketing personalization platform on the market. But what my time at Qubit also made clear to me was that, when it came down to it, what really motivated me to get up in the morning, learn all these new technologies and still be wildly excited to come into work in the morning twenty years later was: - using data and analytics to find new insights and uncover new opportunities in a customer’s data set - working with that individual clients, over time, to enable them to find more of those insights and opportunities themselves - find new innovations in analytics technologies and how we deliver projects to make this process cheaper, faster and more likely to be successful - and building a team, and crucially a business, to do all of this at scale and offer a full set of analytics-related consulting services built around modern analytics tools and delivery techniques Which is why after two years away from the consulting business and two enjoyable, rewarding and enlightening years working on the other side of the data and analytics industry I’m now launching my new consulting company, [MJR Analytics](https://mjr-analytics.com/); and I hope to be working with many of you as clients or members of our team over the coming months and years. --- # Date Partitioning and Table Clustering in Google BigQuery (and Looker PDTs) Canonical: https://rittmananalytics.com/blog/2018/08/27/2018-08-27-2018-08-27-date-partitioning-and-table-clustering-in-google-bigquery-and-looker-pdts URL: https://rittmananalytics.com/blog/2018/08/27/2018-08-27-2018-08-27-date-partitioning-and-table-clustering-in-google-bigquery-and-looker-pdts Description: Google BigQuery is a data warehousing-orientated “table-as-a-service” product from Google Cloud Platform that, like Oracle Exadata, optimizes for full-table scans rather than selective row access and stores data organized into… Author: Mark Rittman Published: 2018-08-27T10:27:13+00:00 Modified: 2018-08-27T10:27:13+00:00 --- Google BigQuery is a [data warehousing-orientated “table-as-a-service” product](https://medium.com/p/39f3c6308d55) from Google Cloud Platform that, [like Oracle Exadata, optimizes for full-table scans](https://blogs.oracle.com/smartscan-deep-dive/the-beginners-guide-to-oracle-table-scans) rather than selective row access and stores data organized into columns, rather than rows, to align better with filtering and aggregation workloads associated with data warehousing workloads. BigQuery charges by amount of data stored in your tables and the data you’ve read in executing SQL queries against those tables, so [BI tools such as Looker that work efficiently with BigQuery](https://looker.com/solutions/google-bigquery) only request the columns they need when querying BigQuery datasets [rather than running a SELECT(\*)](https://cloud.google.com/bigquery/docs/best-practices-costs#avoid_select) and throwing away what’s not needed. To illustrate this let’s run a query that requests all columns (“SELECT (\*) FROM …”) from a BigQuery table, and as you can see from the screenshot below it’s reading through all 1.47GB of table data to return that full set of columns to the user. If the users’ query only really needed just two of those columns, requesting just those brings down the amount of data read to just under 10MB as most of of that table’s data is stored in columns of data that aren’t needed for the query. BigQuery historically has supported table partitioning based on the [date you loaded data into the table](https://cloud.google.com/bigquery/docs/creating-partitioned-tables) which sounds great until you realise that it’s the transaction date, not the ingest date, that most of your user’s queries filter against. You could also use [table decorators](https://cloud.google.com/bigquery/table-decorators) in Legacy SQL to point to the particular day partition your data was stored within but this only went back for a maximum of seven days and required your query tool to support this non-standard feature of SQL; earlier in this year though Google [introduced a more flexible form of date partitioning](https://cloud.google.com/bigquery/docs/partitioned-tables) as a beta release feature that allows you to choose the date column your table would be partitioned by, and more recently introduced a feature called table clustering that stores data within a table sorted by the columns you most commonly filter on when running queries against it. To show how date partitioning and table clustering work, I’ll start by running a query to return just a month’s data from the five years of data held within my table; as you can see in the screenshot below, BigQuery performs a full table scan and reads through all 1.37 GB of data in the table to return just the month of data my query requested. Standard SQL [now supports DDL commands](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-definition-language) such as CREATE TABLE and CREATE TABLE … AS SELECT, along with a [PARTITION BY](https://cloud.google.com/bigquery/docs/creating-column-partitions) clause that lets you specify a timestamp or date column to partition the table by. I’ll use these new features to create a copy of that same table, this time partitioned by the timestamp column I’m filtering on in my query predicate … … and the DDL statement fails. What’s happened there then? Turns out that [BigQuery tables are limited to 2500 partitions](https://stackoverflow.com/questions/44118590/bigquery-partitioning-data-past-2000-limit) for the moment, with any one single load operation limited to 2000 (hence the error) and with partitioning limited to just date and timestamp columns and partitions a day in length it means any one table can only hold around five years worth of data, beyond that you’ll need to create multiple date partitioned tables and UNION them together through a SQL view. For now though I load my table with just five years of data and then re-run the query that requests a single day from that overall five years; now BigQuery has only read and processed 57 MB of data and it’d be a fraction of that if I only requested the two columns I needed, not all columns from that table. But what about queries that filter against the other columns in this table? We can’t set up table partitioning on STRING, INTEGER or any other type of column datatype so my original query if re-run against the date partitioned table reads just as much data as it did before. What we could do is re-create the table with its data pre-sorted by those two particular columns using another new feature called table clustering, so that queries that filter against those columns find the blocks of data they’re looking for faster and can skip completely the ones that don’t. If like me you’re more familiar with databases such as Oracle, “table clustering” is all about [storing data from tables sharing a common cluster key together in the same data blocks](https://docs.oracle.com/cd/E11882_01/server.112/e40540/tablecls.htm#CNCPT608), so that queries against that group of tables filtering on that key return data faster. Table clustering in BigQuery is more analogous to [loading regular Oracle tables using data from a pre-sorted file](https://docs.oracle.com/database/121/SUTIL/GUID-571EC032-0C25-483A-BC0D-2C27A7B64D64.htm#SUTIL1316) and comes with the same benefits and limitations; in BigQuery’s case it takes care of the pre-sorting and table configuration for you but the same limitations still apply around how you filter the table and what happens when you load more data afterwards. Let’s set-up a clustered table now that stores its data ordered by the two columns used in the query I ran a few moments ago. Now when I run a query filtering on those columns against this partitioned, clustered table the amount of data read goes down compared to before, and results are returned a bit faster; if I included the partition key column in the query as well, returning just a few days’ data, it’d be faster still. But queries that filter using any of the other columns don’t benefit from this clustering and moreover, [the benefits of sorting the data initially loaded into a clustered table are lost over time as you load more (unsorted) data into it](https://cloud.google.com/bigquery/docs/clustered-tables), meaning that to really benefit from clustering a table you have to rebuild it regularly using a CREATE TABLE … AS SELECT. Table clustering in BigQuery is nice to have but similar to pre-sorting a direct path load into Oracle database tables, it’ll take a lot of rebuilding and careful querying to get the most benefit from and with the size of most BigQuery tables, I doubt that rebuilding will happen much in-practice. BI tools such as Looker can make use of table partitioning and clustering in queries straight away as no changes are required in the query SQL you write, everything is handled in the table definition. Where you might want to set up partitioning yourself as a Looker developer is for the [persistent derived tables (PDTs)](https://docs.looker.com/admin-options/database/pdts) that Looker can create to materialize the results of a view you define in your Looker model to derive a dimension attribute using a subquery, for example to calculate the sequence number of a users’ order for retention analysis for an eCommerce dashboard as shown in the screenshot below. Looker has for some time come with database-specific settings for particular source database types [such those used for Redshift](https://docs.looker.com/data-modeling/learning-lookml/derived-tables#indexing_pdts) used in the example above, and now supports date partitioning for persistent derived tables through a new partition\_keys setting as announced in this [recent Looker forum post](https://discourse.looker.com/t/announcing-date-time-partitioned-pdts-for-google-bigquery/7132). Finally, if you’re interested in how partitioning is developing as a feature within BigQuery, and some of the edge-cases and requests for additional functionality that users of partitioning are asking for, [this feature request on the BigQuery issue tracker](https://issuetracker.google.com/issues/35905817) is useful to read-through. Extending support to more than 2500 partitions seems to be the most popular request along with allowing integer and string datatype columns to be used for the partition key, but also look out for issues around re-loading data into historic partitions and the cost and work involved in rebuilding large tables to re-cluster its data or apply partitioning for the first time. --- # Oracle Big Data Cloud, Event Hub and Analytics Cloud Data Lake Edition pt.3 Canonical: https://rittmananalytics.com/blog/2018/06/02/2018-06-02-2018-06-02-oracle-big-data-cloud-event-hub-and-analytics-cloud-data-lake-edition-pt-3 URL: https://rittmananalytics.com/blog/2018/06/02/2018-06-02-2018-06-02-oracle-big-data-cloud-event-hub-and-analytics-cloud-data-lake-edition-pt-3 Description: In this series of three blogs on Oracle Analytics Cloud Data Lake Edition I’ve setup an object store data lake in Oracle Cloud using Oracle Big Data Cloud and Oracle… Author: Mark Rittman Published: 2018-06-02T01:52:05+00:00 Modified: 2018-06-02T01:52:05+00:00 --- In this series of three blogs on Oracle Analytics Cloud Data Lake Edition I’ve [setup an object store data lake in Oracle Cloud](https://medium.com/mark-rittman/oracle-big-data-cloud-event-hub-cloud-and-analytics-cloud-data-lake-edition-pt-1-84961cd4274f) using Oracle Big Data Cloud and Oracle Storage Cloud, and ingested streams of real-time event data from IoT and social media sources into Oracle Cloud’s object storage service using Oracle Event Hub Cloud Service. The event-stream data I staged into Storage Cloud was then copied into parquet files on HDFS and then presented out to BI and ETL tools through Big Data Cloud’s Thrift Server interface, so that now I’m ready, after a [short diversion into defining the data engineer role](https://medium.com/mark-rittman/oracle-big-data-cloud-event-hub-and-analytics-cloud-data-lake-edition-pt-2-9f6b004dc0d1) that would typically work with this new product edition, to start exploring some of Oracle Analytics Cloud Data Lake Edition’s new data flow and predictive model preparation features. The diagram below shows where OAC Data Lake Edition fits into my project architecture, performing the tasks of transforming and enriching the incoming dataset and then presenting my at-scale data out to end-users for analysis using OAC Data Lake Edition’s Data Visualization features. Looking at the homepage within OAC Data Lake Edition I can see my two Hive tables listed within the dataset catalog, alongside other datasets I’d uploaded directly into OAC. This visual catalog of available datasets is also the new homepage interface that OBIEE12c 12.1.0.4.0 now adopts, with both cloud and on-premises versions of Oracle’s BI tools now relegating the old “Answers” homepage to something you have to dig around and specifically look for in favour of this more self-service Data Visualization starting page. I’ll have to write an article on Answers and how powerful its interface is, and the full dimensional model it exposes from the Oracle BI Repository, in a blog post sometime in the future as it’s almost in danger of getting forgotten about. Moving on though, the first transformation I need to do on all the incoming datasets is to take the timestamp column in each table and convert it to a format that OAC recognises as a valid TIMESTAMP datatype format, then convert those columns to TIMESTAMPs so that DV can automatically enable time-series analysis by day, month, quarter, hour and so on. I do that using a feature that’s also present in OAC Standard Edition, the lightweight data preparation interface that’s presented to users when they first add a new data source into OAC’s dataset catalog, shown in the screenshots below. Where OAC Data Lake Edition gets really interesting right now both in terms of differences vs. the on-premises versions of OBIEE I used to use, and in terms of it’s “data engineering” potential, is with a feature called *Data Flows*. Most self-service BI tools now have a basic data loading and data preparation capability today with [Tableau Data Prep](https://www.tableau.com/products/prep) being one of the latest examples. Designed to handle more complex data prep use-cases than basic datatype changes and field-splitting, they give end-users the ability to do this type of work themselves rather than trying to do it in Excel or handing the work off to the IT department and having to wait days or weeks to get that data back. Data Flows are a feature that’s been introduced since the original on-premises version of OBIEE12c that I last used when working out in consulting, and provide you with what’s effectively a lightweight, multi-step ETL tool that executes transformations using the BI Server’s Model Extension feature, introduced back when OBIEE12c first came out as the mechanism to enable on-the-fly data mashups between server-side and user-uploaded datasets. Looking at the transformation operators available in OAC Data Lake Edition v4 there’s quite a few that apply to data lake and data engineering-type workloads including running Python statistical analysis scripts and predictive model training and model build; there’s also an operator for creating an Essbase Cube, with Essbase in this instance positioned as a fast ad-hoc analysis back-end for use with the data visualization part of OAC. For now though there’s two transformation tasks I want to do with my Hive datasets; first, enrich the incoming social media data by analyzing the sentiment in each tweet and then writing the data plus this sentiment tagging back to the Oracle Big Data Cloud environment, so that I can then turn those sentiment tags into a score and create a data visualization showing who sends me the most tweets and how crazy they are overall. The second data enrichment I wanted was on some Strava cycling workout data I’d uploaded directly into OAC using the CSV file upload facility; using the model train and build Data Flow operators I defined a model to predict how many “kudos”, the Strava equivalent to Facebook “likes”, I’d get for a given cycle workout with a number of different variables available to the model in order to make the prediction — for example, distance and elevation gain, map location, effort expended and so on. Then, after running the model build step and looking at the predicted values and the actual ones for the remainder of the dataset not used for model training, you can see the predicted kudos values are fairly in-line with the ones I actually recorded for those rides. Another feature that’s now in Oracle Analytics Cloud is automated data diagnostics, or *Explain*. Explain uses machine-learning libraries and that same model extension/XSA BI Server framework to help users quickly understand the value distribution and statistically correlated driving factors for a particular dataset, and learn which segments or cohorts have the highest predictive significance. Enabled by a bunch of extensions to BI Server logical SQL I used the feature first on the sentiment scoring I’d performed earlier on, and then on the steps data I’d brought into Oracle Big Data Cloud from my Fitbit device, after converting the numeric step counts into a text attribute by bucketing its values into low, medium and extreme bucket values. This is pretty powerful stuff, with automated understanding and context-gaining about new datasets being one of the most user-enabling features I’ve seen arrive recently in BI tools with the best example of this being [BeyondCore](https://vimeo.com/160139406), now part of Salesforce Einstein. OAC lets the user pick the most useful of the Explain facts and driver insights and publish them to a Data Visualization dashboard like the one below, showing the most predictive and significant variables in my dataset that influence the steps I take each day. Which leads neatly to the final “data at-scale” feature in OAC, the Data Visualization feature that in my case is querying the ingested, transformed and now enriched datasets I’ve got running on my Oracle Big Data Cloud instance alongside Oracle Event Hub Cloud and Oracle Analytics Cloud Data Lake Edition. Thank you once again to the [Oracle ACE Director program](http://www.oracle.com/technetwork/community/oracle-ace/index.html) for providing access to Oracle Analytics Cloud Data Lake Edition, Oracle Big Data Cloud and Oracle Event Hub Cloud services over the past few weeks. If you’re looking to try these new services out there’s [free trials available](https://cloud.oracle.com/tryit%5C) for most of Oracle’s Cloud products and many of the new features are also available in [Oracle Data Visualization Desktop 12c](http://www.oracle.com/technetwork/middleware/oracle-data-visualization/downloads/oracle-data-visualization-desktop-2938957.html) and [Oracle Business Intelligence 12c](http://www.oracle.com/technetwork/middleware/bi-enterprise-edition/downloads/default-4441820.html), both of which can be downloaded for training and evaluation under the OTN license scheme. Wrapping-up this three part series on Oracle Analytics Cloud Data Lake Edition and Oracle Big Data Cloud I’d like to go back to the two (serious) questions I asked myself at the end of the previous post: 1. Has OAC Data Lake Edition got anything actually to do with data lakes, and is it a useful tool for aspiring Oracle technology data engineers? 2. How does it compare to my old favourite Oracle big data product [Oracle Big Data Discovery](https://www.oracle.com/uk/big-data/big-data-discovery/index.html), officially still available and not quite dead yet but existing in some strange zone where the on-premises version stopped getting updates a while ago and the cloud version is for sale but you can’t buy it unless you know the right person to ask and he’s actually gone to Cloudera So has Oracle Analytics Cloud Data Lake Edition got much to do with actual “data lakes”? Well … it integrates with Oracle Big Data Cloud and apparently comes with an option to run those data flow transformation in Big Data Cloud’s Apache Spark environment, though to be fully-transparent I didn’t see that as an option when doing my evaluation so can’t comment on how well or not that works. Like Oracle Big Data Discovery before it, OAC Data Lake Edition makes you structure your incoming event stream data with Hive table metadata before you can work with it, but that’s actually fairly standard practice with most data visualization tools that work with Hadoop and data lake environments. Having Essbase in this product package, alongside the data lake functionality, did make me scratch my head a bit and wonder, “why?” — data lakes and Essbase are about as opposite as you can get in terms of target users and use-cases and I think this Data Lake Edition is as much about creating a product package and price point that’s mid-way between the OAC Standard and Enterprise Edition. But there is some logic to having Essbase in this edition; it provides a set of easy-to-use loading and preparation tools for Essbase making it easier for customers new to that product to start using it, and Essbase with its latest hybrid ASO/BSO storage format is surprisingly scalable and blindingly-fast to query, a great potential back-end for enabling data analysis “at-scale” using Oracle Analytics Cloud’s data visualization features. I also get the feeling that this initial v4 version of OAC Data Lake Edition is more of an initial first-cut release to get something out to customers, establish the product package and validate the roadmap and market assumptions. Oracle Analytics Cloud v5 isn’t too far off and I’d expect incremental improvements and new features in areas such as natural language processing and machine learning built-into the developer experience; I wouldn’t be surprised to see [Oracle Big Data Preparation Cloud](https://cloud.oracle.com/bigdatapreparation) making its way into the product given its obvious fit and overlap with Data Lake Edition’s data prep features. But where I really see an interesting future for OAC Data Lake Edition is when it starts to integrate product features and the development team from [Oracle’s recent acquisition of Sparkline Data](https://www.oracle.com/corporate/acquisitions/sparklinedata/index.html). I came across [SNAP](http://sparklinedata.com/product/), Sparkline‘s platform for building OLAP-style dimensional models over data lakes and cloud object storage layers about a year ago when researching analytics platforms at Qubit and quite frankly, it’s as revolutionary in terms of todays data lake analytics market as OBIEE ([or the nQuire Server](https://www.rittmanmead.com/blog/2007/09/a-potted-history-of-oracle-bi-suite-enterprise-edition/)) was back in 1995 with its virtual data warehouse over application and data warehouse sources. Take these two slides from the Sparkline website and imagine them as the future of Oracle Analytics Cloud analyzing event-streams and big data cloud-hosted datasets… and you can see why I’m keen to see where Oracle Analytics Cloud Data Lake Edition goes over the next couple of years. I’m speaking on OAC Data Lake Edition at [ODTUG KScope’18 in Orlando](https://kscope18.odtug.com/) in just a couple of weeks time so come along if you’re there, it should be an interesting talk. --- # Oracle Big Data Cloud, Event Hub and Analytics Cloud Data Lake Edition pt.2 Canonical: https://rittmananalytics.com/blog/2018/05/28/2018-05-28-2018-05-28-oracle-big-data-cloud-event-hub-and-analytics-cloud-data-lake-edition-pt-2 URL: https://rittmananalytics.com/blog/2018/05/28/2018-05-28-2018-05-28-oracle-big-data-cloud-event-hub-and-analytics-cloud-data-lake-edition-pt-2 Description: A couple of weeks ago I posted a blog on Oracle Analytics Cloud Data Lake Edition and the Oracle Cloud big data products you’d typically use to create a cloud-hosted,… Author: Mark Rittman Published: 2018-05-28T18:26:02+00:00 Modified: 2018-05-28T18:26:02+00:00 --- ![Oracle Big Data Cloud, Event Hub and Analytics Cloud Data Lake Edition pt.2](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2Ff8fdf-60ae5a44-a8b2-4c2e-949c-482e148224ad_643x286.png%3Fssl%3D1) A couple of weeks ago I posted a blog on [Oracle Analytics Cloud Data Lake Edition and the Oracle Cloud big data products](https://medium.com/mark-rittman/oracle-big-data-cloud-event-hub-cloud-and-analytics-cloud-data-lake-edition-pt-1-84961cd4274f) you’d typically use to create a cloud-hosted, object storage data lake. In this and the final post in the series I’m going to look at some of the data transformation and predictive model building features that come with the Data Lake Edition of OAC and extend the base capabilities in this area that OAC Standard Edition comes with, and the data engineering role that uses the data lake this Oracle Analytics Cloud product packaging option is designed for. As a reminder, OAC Standard Edition is more-or-less Oracle Data Visualization running as a single-tenant cloud-hosted VM (cluster) that’s either customer managed, or Oracle-managed if you go for the new [Autonomous Analytics Cloud](https://www.oracle.com/uk/autonomouscloud/index.html) that interestingly is [charged at a lower hourly rate](https://www.oracle.com/uk/autonomouscloud/index.html) than the customer-managed version; presumably it costs Oracle less to run OAC when it’s doing all the administration rather than letting customers control the settings, which makes sense. It does make you wonder when all this VM nonsense will go away though as well as the distinction between OAC and the DVCS/BICS managed service cloud analytics products, with all of these services just becoming packaging options for a multitenant service partitioned by customer instance as the likes of Looker and Google Cloud Platform do today. But for now, Oracle Analytics Cloud Data Lake Edition is described on the [product homepage](https://cloud.oracle.com/en_US/oac/pricing) as having the following high-level capabilities compared to OAC Standard Edition: ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2Ff8fdf-60ae5a44-a8b2-4c2e-949c-482e148224ad_643x286.png%3Fssl%3D1) - Accelerated analysis via Essbase - and support for creating scalable, shareable high performance data flows. whilst the [product documentation](https://docs.oracle.com/en/cloud/paas/analytics-cloud/acsgs/what-is-oracle-analytics-cloud.html#GUID-ED616A54-9901-4829-A62B-514730B5416B) states that it includes all the OAC Standard Edition features, plus: - Explore and replicate data - Create scalable data flows including ingestion, transformation, and persistence - Execute data flows by using Apache Spark or Oracle Database Cloud with two edition-specific scenarios listed: - Explore Data Lakes, described as *“You can create data flows to visualize data in your data lakes”*, and - “Replicate Data in Data Lakes”, explained as *“You can replicate data from Oracle Service Cloud (RightNow) into Oracle Big Data Cloud Service, Oracle Database Cloud, and Oracle Database. For example, you can replicate data from customer service records in Oracle Service Cloud into your data lake on Oracle Big Data Cloud Service and then perform further data preparation and analysis with data flows and data visualization projects.”* I’m guessing most readers won’t have heard of [Oracle Service Clou](https://cloud.oracle.com/service-cloud)d but for Oracle Cloud customers it’s quite a useful initial integration, as Service Cloud is Oracle’s cross-channel digital customer call centre application that brings together customer service interactions across web, social media and offline service channels. More interestingly though and you may have missed it back at the start but Data Lake Edition is the edition that ships [Essbase](http://www.oracle.com/technetwork/middleware/essbase/overview/index.html), Oracle’s multi-dimensional OLAP server, in what almost seems like an afterthought right now but will become, I suspect, a more significant defining feature of this product package as time goes on. For the purposes of this series of blog posts though, what I’m really interested in are the data engineering features in OAC Data Lake Edition — but before I start looking into what they might be, I guess I’d better explain what I mean by a *data engineer*. * * * I first came across the term *data engineer* in [a blog written by Curt Monash](http://www.dbms2.com/2016/07/31/terminology-data-scientists-vs-data-engineers/) — still the #1 person I’d love to have come on the Drill to Detail Podcast, if you happen to read this Curt — where he wrote: > “I learned some newish terms on my recent trip. They’re meant to solve the problem that “data scientists” used to be folks with remarkably broad skill sets, few of whom actually existed in ideal form. So instead now it is increasingly said that: > – “Data engineers” can code, run clusters, and so on, in support of what’s always been called “data science”. Their knowledge of the math of machine learning/predictive modeling and so on may, however, be limited. > – “Data scientists” can write and run scripts on single nodes; anything more on the engineering side might strain them. But they have no-apologies skills in the areas of modeling/machine learning.” I think it’s probably true to say that the term really started to enter people’s consciousness, at least the sorts of people I hang around with, when Maxime Beauchemin from Airbnb wrote his seminal blog [“The Rise of the Data Engineer”](https://medium.freecodecamp.org/the-rise-of-the-data-engineer-91be18f1e603) where he introduced this new discipline as: > “Like data scientists, data engineers write code. They’re highly analytical, and are interested in data visualization. > Unlike data scientists — and inspired by our more mature parent, **software engineering** — data engineers build tools, infrastructure, frameworks, and services. In fact, it’s arguable that data engineering is much closer to software engineering than it is to a data science. > In relation to **previously existing roles**, the data engineering field could be thought of as a superset of **business intelligence** and **data warehousing** that brings more elements from **software engineering**. This discipline also integrates specialization around the operation of so called “big data” distributed systems, along with concepts around the extended Hadoop ecosystem, stream processing, and in computation at scale. > In smaller companies — where no **data infrastructure** team has yet been formalized — the data engineering role may also cover the workload around setting up and operating the organization’s data infrastructure. This includes tasks like setting up and operating platforms like Hadoop/Hive/HBase, Spark, and the like.” Maxime’s blog struck a chord with me as that’s pretty much how I’d describe what I do, or at least it’s the work I most enjoy doing; building big data systems to support BI and data warehousing workloads where my knowledge of how Hadoop, cloud and distributed data platforms complements the knowledge that other colleagues have around data science, machine learning and statistical analysis though importantly, I know enough about those areas to be able to deploy the models they build (or as we say over in the UK at least, I know enough about them to be dangerous). [I interviewed Maxime about his blog and the topic of data engineering](https://www.drilltodetail.com/podcast/2017/5/15/drill-to-detail-ep26-airflow-superset-the-rise-of-the-data-engineer-with-special-guest-maxime-beauchemin) on the Drill to Detail Podcast last May and if you’ve not done so already I’d recommend you give the episode a listen; for me though, having just spent the last eighteen months working in the London tech startup scene, my definition of a data engineer would be someone who’s just out of university and has an ironic beard, thinks he’s made ETL tools obsolete but is slowly, painfully, recreating one and is now just on the brink of discovering code templating, deployment contexts, change control and data lineage… ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2Fb486c-6e959b58-c002-455d-871f-915ae6642f13_800x225.png%3Fssl%3D1) … but earns twice what you earn and gets all the girls. Because, data engineers are even more essential than data scientists if you’re serious about building a data lake, and data lakes are the new data warehouses. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2F5e5b9-6534070f-35cf-484b-a656-2c1de5611f21_741x388.png%3Fssl%3D1) * * * Which bring us back to Oracle Analytics Cloud Data Lake Edition and these three blog posts. The first part of this three part series talked about Oracle Big Data Cloud and products such as Oracle Event Hub Cloud — Dedicated (who names these products at Oracle? Seriously?) and in this second I seem to have got rather diverted by trying to define what Oracle Analytics Cloud Data Lake Edition actually is and then even more so by defining the supporting role of a *data engineer;* in the final post tomorrow I’ll take a look at some of the data transformation, model definition and deployment and *data engineering* features in OAC Data Lake Edition and then, finally, conclude by asking myself two questions — OK, three: 1. Has OAC Data Lake Edition got anything actually to do with data lakes, and is it a useful tool for aspiring Oracle technology data engineers? 2. How does it compare to my old favourite Oracle big data product [Oracle Big Data Discovery](https://www.oracle.com/uk/big-data/big-data-discovery/index.html), officially still available and not quite dead yet but existing in some strange zone where the on-premises version stopped getting updates a while ago and the cloud version is for sale but you can’t buy it unless you know the right person to ask and he’s actually gone to Cloudera 3. Will the ironic beard and quiff be enough, or do I need to go full ginger if I’m going to really be taken seriously as a data engineer? ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2F4c0f8-1b5e0431-b5f0-481f-b4a7-c209f1b9091c_619x349.png%3Fssl%3D1) Final part in the series on Oracle Analytics Cloud Data Lake Edition comes later this week. --- # Oracle Big Data Cloud, Event Hub and Analytics Cloud Data Lake Edition pt.1 Canonical: https://rittmananalytics.com/blog/2018/05/10/2018-05-10-2018-05-10-oracle-big-data-cloud-event-hub-and-analytics-cloud-data-lake-edition-pt-1 URL: https://rittmananalytics.com/blog/2018/05/10/2018-05-10-2018-05-10-oracle-big-data-cloud-event-hub-and-analytics-cloud-data-lake-edition-pt-1 Description: Some time ago I posted a blog on what analytics and big data development looked like on Google Cloud Platform using Google BigQuery as my data store and Looker as… Author: Mark Rittman Published: 2018-05-10T00:16:50+00:00 Modified: 2018-05-10T00:16:50+00:00 --- ![Oracle Big Data Cloud, Event Hub and Analytics Cloud Data Lake Edition pt.1](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2F2d5cc-c5a07b9d-df6a-4a7e-b426-37ab97ecb4e6_775x517.png%3Fssl%3D1) Some time ago I posted a blog on [what analytics and big data development looked like on Google Cloud Platform](https://medium.com/mark-rittman/what-bi-development-looks-like-with-bigquery-google-cloud-apis-looker-and-fluent-courtesy-of-the-f9e8c23ba0c6) using Google BigQuery as my data store and Looker as the BI tool, with data sourced from social media, wearable and IoT data sources routed through a Fluentd server running on Google Compute Engine. Overall, the project architecture looked like the diagram below… ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2F2d5cc-c5a07b9d-df6a-4a7e-b426-37ab97ecb4e6_775x517.png%3Fssl%3D1) … and I’ve got more-or-less the same setup running right now, with an additional GCE VM running Confluent Open Source to feed a subset of the event streams into a Druid cluster that I’m using to test out Looker, Superset and Imply for sub-second ad-hoc query analysis use-cases. More on that soon. If you’re a regular reader of this blog you might recall a week or so ago I posted a blog on the new releases of Oracle Business Intelligence Enterprise Edition (OBIEE), Oracle Analytics Cloud (OAC) and Oracle Data Visualization Desktop (DVD) and mentioned a new packaging option for their cloud analytics product, Oracle Analytics Cloud Data Lake Edition. I’ve got a particular interest in what this product might be as I used to use the product it replaces, Oracle Big Data Discovery (BDD), fairly extensively in [Oracle big data analytics](https://www.slideshare.net/rittmanmead/using-oracle-big-data-discovey-as-a-data-scientists-toolkit) and [data lake](https://www.youtube.com/watch?v=yI1v61-oCZE&list=PL6Vx6k9b2zbVLEQFymZ2UbRMvUlaPRCGm&index=0) projects a few years ago. And Oracle Big Data Discovery was — technically at least — a great product. It combined the search and analytics features of Endeca Information Discovery with the scale and data transformation abilities enabled by Apache Spark and Hadoop, but suffered perhaps by being a bit ahead of the market and by not having any obvious integration with the rest of Oracle’s analytics and data management tools. By contrast Oracle Analytics Cloud Data Lake Edition is one of three packaging options for Oracle Analytics Cloud and includes all of the functionality of OAC Standard Edition (Oracle Data Visualization together with basic data preparation tools) as well as itself being a subset of the wider set of analysis, dashboarding and enterprise data modeling features in OAC Enterprise Edition. An equivalent product architecture for ingesting, transforming and analyzing my IoT, wearables and social media data in Oracle Cloud would look something like the diagram below, with the following Oracle Cloud Platform-as-a-Service (PaaS) products used for ingest, storage and analysis: - **[Oracle Event Hub Cloud Service](https://cloud.oracle.com/event-hub)**: Apache Kafka running either customer or Oracle-managed with full access to Kafka’s REST Proxy and Kafka Connect - **[Oracle Big Data Cloud](https://cloud.oracle.com/bigdata)**: Oracle’s new elastically-scalable Hadoop platform running Apache Spark, Ambari and other Hortonworks Data Platform components - **[Oracle Analytics Cloud Data Lake Edition](https://blogs.oracle.com/emeapartnerbiepm/oracle-analytics-cloud-data-lake-edition-available)**: Oracle Data Visualization combined with more extensive lightweight ETL (“data flow”) components, text analytics and machine learning model training and build capabilities ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2F233ba-370f2e7c-a24d-4745-8de4-7f64d1d37eef_800x369.png%3Fssl%3D1) In this example I’m using Apache Hive and Parquet storage as my column-orientated data platform but of course I’ve now also got [Oracle Autonomous Data Warehouse Cloud](https://medium.com/mark-rittman/first-impressions-of-oracle-autonomous-data-warehouse-cloud-acebc82387f4) as an option; I’ll stick with Hive on Oracle Big Data Cloud for now though as this gives me the option to use Apache Spark to transform and wrangle my data and for building machine learning models using SparkML and, via pySpark, Python Pandas. In what’s the first of two posts in this short series I’ll be looking at how the data pipeline is set up, and then in the second post I’ll look at Oracle Analytics Cloud Data Lake Edition in detail focusing on the data transformation, data engineering and data science features it adds beyond OAC Standard Edition. * * * The development environment I put together for this scenario used the following Oracle Infrastructure-as-a-Service (IaaS) and Platform-as-a-Service (PaaS) components: - **Oracle Compute Classic** and **Storage Classic** Services - **Oracle Database Cloud Service**, with the 11g database option - **Oracle Event Hub Cloud Service Dedicated**, with Kafka Connect and REST Proxy nodes - **Oracle Big Data Cloud**, single node with Hive, Spark 2.1, Tez, HDFS, Zookeeper, Zeppelin, Pig and Ambari - **Oracle Analytics Cloud Data Lake Edition** with Self Service Data Preparation, Visualisation and Smart Discovery (aka Oracle DV) The screenshot below from the Compute Classic Service Console shows the various PaaS VMs running on the compute IaaS layer, with other tabs in this console showing network, storage and other infrastructure service usage. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2F58aab-56777519-0701-44e4-9c61-bb3e49b9b439_800x433.png%3Fssl%3D1) The order in which you install the services is important if you want to associate the various products together correctly, and if like me you’re using a trial account you’ll need to plan carefully to ensure you keep with the various quota limits that Oracle Cloud imposes on trial identity domains that stopped me, for example, allocating the usual two OCPUs to the main Oracle Event Hub Kafka server if I wanted to run the rest of the stack at the same time. Associating two services together, for example Oracle Event Hub and Oracle Big Data Cloud, connects the two services together automatically in the identity domain network and makes using them together much simpler but unless Event Hub Cloud is provisioned and available when you come to install Big Data Cloud you can’t go back and associate them afterwards, but more annoyingly if you decide you want to use Event Hub Cloud and associate it with Big Data Cloud but since then you’ve provisioned Oracle Analytics Cloud Data Lake Edition and associated Big Data Cloud with that you have to unwind the whole provisioning process and start again with Event Hub Cloud if you want to connect them all together properly. And forget deleting that Database Cloud Service you associated with Oracle Big Data Cloud and then forgot about as you can’t delete services that other services are associated with. Provisioning each of the services involves giving the service instance a name, assigning storage buckets and OCPU counts to the various cluster nodes you request, and at key points selecting previously provisioned and now running services for association with the one you’re now provisioning. The screenshots below show the three-stage provisioning service for Event Hub Cloud Service — Dedicated: ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2F97d15-c7185fcc-9cc6-4771-b995-fd085b097c89_800x558.png%3Fssl%3D1) Provisioning a new Event Cloud Service — Dedicated cluster and the order in which I provisioned my data lake services, and the important options I chose to make it all work together and within quota, were as follows: 1. First ensure you have access to Oracle Cloud, and then the various IaaS services: Oracle Cloud Infrastructure Compute Classic, Oracle Cloud Infrastructure Object Storage Classic and Oracle Identity Cloud Service 2. Provision Oracle Database Cloud Service to store the various Fusion Middleware RCU schemas; in my case I chose Oracle Database 11gR2 as the database type as it avoids the complexity around CDBs and PDBs you get with the 12c database release 3. Then provision Oracle Event Hub Cloud — Dedicated with one OCPU for the main Kafka VM, one for the REST Proxy VM and another one OCPU for the Kafka Connect VM. For a real deployment you’d want at least two OCPUs for the Kafka service VM but using just the one kept me within my overall OCPU quota limit when installing the rest of the stack 4. Next step is to provision Big Data Cloud with a single node with the minimum 2 OCPUs, the Full deployment profile and version 2.1 of Apache Spark as that’s the version OAC Data Lake Edition insist on in the next step. When prompted, choose the option to associate Event Hub Cloud and Database Cloud with Big Data Cloud as you won’t get the option to do this again after the initial service provision; once provisioned, open-up the TCP port for Ambari (8080) to the public internet so that OAC in the next step can associate with it — provisioning for OAC failed for me every time until I looked through the provisioning logs and spotted this as the issue 5. Finally, provision Oracle Analytics Cloud and choose Data Lake Edition as the package option, again in my case assigning a single OCPU and selecting Data Lake Edition as the software version At that point if you then bring up the Cloud Services dashboard and review the services together for the first time, it’ll look something like this: ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2Fd3a98-89783f1b-405f-465c-8722-9df32bde38fe_800x315.png%3Fssl%3D1) Oracle Data Lake stack within Cloud Services Dashboard Now it’s time to ingest some data and land it into Oracle Big Data Cloud. * * * The streaming IoT, wearables and social media comms data that I’ll be ingesting into Big Data Cloud will be coming in from the public internet over TCP, and I’ll also want to connect to Event Hub Cloud from my desktop using tools such as [Kafka Tool](http://www.kafkatool.com/) so an addition configuration step I’ll do before setting up anything else is to open-up Event Hub Cloud’s Kafka broker endpoint to the public internet using the Access Rules menu item in the Event Hub Cloud console. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2Fb1f10-82053f8e-84bd-4ad3-af1b-24be93ecb3a6_757x488.png%3Fssl%3D1) Now I can see the Kafka service and the default topics that Event Hub Service created for me in Kafka tool. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2F39d96-ebe2272d-2971-4907-92d1-bc2f4fb0a4dc_800x323.png%3Fssl%3D1) I can either then use Kafka tool to create a new topic to start receiving the first of my data streams, the IoT device event data coming out of Samsung SmartThings, or create the topic as by defining a new Event Hub Cloud service from within the Event Hub Cloud Service — Dedicated console (confusing, but that’s how Kafka topics are named within Event Hub Cloud) ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2F2f7e1-468eff6c-69d8-4570-a2f6-17e4a558ea28_800x372.png%3Fssl%3D1) Then it’s just a case of directing the stream of IoT event data to the public Kafka broker endpoint exposed by Event Hub Cloud Service — Dedicated and then, after a short while, checking the metrics for the new Kafka topic that I setup to receive this incoming streaming data. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2Fa800d-c563bd13-009d-4f18-bb22-e32b872ed32c_800x640.png%3Fssl%3D1) Getting the data off the Kafka topic and into a Hive table on the Big Data Cloud instance involved the following steps, using Oracle Cloud Infrastructure Object Storage Classic as the intermediate staging layer together with Event Hub Kafka Connect’s OCS Sink Connector: 1. Configure Event Hub Kafka Connect OCS Sink Connector to push topic events to Oracle Cloud Infrastructure Object Storage Classic (OCS) 2. Using Zeppelin notebook provided by Big Data Cloud Console, create a CRON job that copies those events across to HDFS storage 3. Create Hive external tables with location clauses that point to the directories I’ve copied the event files into from OCS Then, when I go and log into OAC Data Lake Edition and connect to the Hive Thrift Server on the Big Data Cloud instance I can see the Hive tables I’ve just created, and the data that’s now streaming through from my two initial sources via the Kafka service and Kafka Connect running on Event Hub Cloud Service — Dedicated. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2F29838-f16f7123-bf4a-48c7-bc0b-55e6612f1914_800x639.png%3Fssl%3D1) In the second half of this two-post series I’ll go deeper into OAC Data Lake Edition and see how its additional transformation and analysis capabilities stack-up against OAC Standard Edition, and also see how it compares to the Oracle Big Data Discovery tool its looking to eventually replace. --- # Using Looker Data Actions to Make Monzo Spend Analysis More Interactive … and Actionable Canonical: https://rittmananalytics.com/blog/2018/05/01/using-looker-data-actions-to-make-monzo-spend-analysis-more-interactive-and-actionable URL: https://rittmananalytics.com/blog/2018/05/01/using-looker-data-actions-to-make-monzo-spend-analysis-more-interactive-and-actionable Description: The other week I posted a blog about analyzing the bank transaction data Monzo make available to customers via their developer API using Looker, the BI tool I use day-to-day… Author: Mark Rittman Published: 2018-05-02T00:55:24+01:00 Modified: 2018-05-02T00:55:24+01:00 --- The other week I posted a blog about [analyzing the bank transaction data Monzo make available to customers](https://medium.com/mark-rittman/monzo-bigquery-looker-and-fintech-the-other-london-tech-startup-scene-554814099154?source=collection_home---4------5----------------) via their developer API using [Looker](https://looker.com/), the BI tool I use day-to-day in my role at Qubit as Analytics Product Manager. One of the dashboards I’ve created shows me how spend looks across a number of merchant categories, compares their average transaction values and higlights the one I visit the most ([The Ginger Dog](http://thegingerdog.com/) in Brighton, thoroughly recommended) and plots each of the them on a map so I can remember where they’re each located. If I click on one of the map icons on the **Spend by Location** visualization Looker then pops-up a dialog showing me the individual column values for this particular merchant — the Hardwood Arms in Fulham, one visit and total/average spend amounts of £75 exactly. As well as tagging your transactions with metadata from the [Foursquare web service](https://foursquare.com/) Monzo also adds the Twitter handle for each merchant along with their website URL; I’ll start to make this dashboard a bit more interactive then by adding these as **link** parameters to my **merchant\_name** dimension definition in my LookML view definition. dimension: merchant\_name {group\_label: “Monzo Transactions”label: “Merchant Name”type: stringsql: ${TABLE}.merchant\_name ;;link: {label:”Merchant Website”url:”[https://](https://){{ fluentd\_monzo.merchant\_metadata\_website.\_value }}”icon\_url: “[https://www.google.com/s2/favicons?domain=google.com”](https://www.google.com/s2/favicons?domain=google.com”);}link: {label:”Merchant Twitter Profile”url:”[https://twitter.com/](https://twitter.com/){{ fluentd\_monzo.merchant\_metadata\_twitter\_id.\_value }}”icon\_url: “[https://www.google.com/s2/favicons?domain=twitter.com”](https://www.google.com/s2/favicons?domain=twitter.com”);} Monzo also provides [Google Places API](https://developers.google.com/places/) and Foursquare API merchant metadata lookup codes along with each transaction record, so I add two more link parameters to call these two web services. link: {label:”View in Google Maps”url: “[https://www.google.com/maps/search/?api=1](https://www.google.com/maps/search/?api=1) &query \={{ fluentd\_monzo.merchant\_address\_latitude.\_value }},{{ fluentd\_monzo.merchant\_address\_longitude.\_value }} &query\_place\_id \={{ fluentd\_monzo.merchant\_metadata\_google\_places\_id.\_value }}”icon\_url: “[https://www.google.com/s2/favicons?domain=maps.google.com”](https://www.google.com/s2/favicons?domain=maps.google.com”);}link: {label:”View in Foursquare”url: “{{ fluentd\_monzo.merchant\_metadata\_foursquare\_website.\_value }}”icon\_url: “[https://www.google.com/s2/favicons?domain=foursquare.com”](https://www.google.com/s2/favicons?domain=foursquare.com”);} Now when I click on the name of a merchant within any data visualization using that data point I get a menu of links I can click on for more information. Clicking on the **View in Google Maps** link passes across the merchant’s latitude, longitude and Google Places API key to the Google Maps web service, and Looker then opens a new browser tab and gives me all the information I’m looking for using the Google Places web service. All of this is good to know but it’s only really of value if I can make this data actionable, for example by linking it to the Google Tasks web service I use to record personal, non-project related tasks and reminders. To do this I’m going to use [Looker’s Data Actions](https://looker.com/platform/actions) feature together with [Zapier’s integration with Google Tasks](https://zapier.com/blog/google-tasks-guide/) to tie it all together. First, I create a Zapier Zap that listens for a webhook call and then uses the parameter values I provide from the Looker data action to to create a new Google Task. [This article](https://discourse.looker.com/t/using-zapier-with-data-actions/5002) on the Looker Discourse website goes into more detail on how to set up Zapier webhooks integration with data actions if you’d like to set up something similar yourself. action: {label: “Add Google Task”url: “[https://hooks.zapier.com/hooks/catch/XXXXXX/fco8nm/”icon\_url](https://hooks.zapier.com/hooks/catch/XXXXXX/fco8nm/%E2%80%9Dicon_url): “[https://www.google.com/s2/favicons?domain=tasks.google.com”form\_param](https://www.google.com/s2/favicons?domain=tasks.google.com%E2%80%9Dform_param): {name: “Task Name”type: stringrequired: yesdefault: “{{ merchant\_name }}”}form\_param: {name: “Due Date”type: stringrequired: yes}form\_param: {name: “Action”type: textareadefault: “Book table at {{ merchant\_name }}”}form\_param: {name: “Urgency”type: selectoption: {name: “Low”label: “Low”}option: {name: “High”label: “High”}}} Then I add an action parameter into my merchant LookML dimension definition that defines a form with default values, allows me to amend those values if needed and then HTTP POSTs them as the payload to the Zapier web service. Now, another menu link item is displayed to allow me to create Google Task items when I click on merchants in a Monzo transactions dashboard, like this: When I click on that link a dialog is then shown, giving me the option to amend the task title and details and add a due date and urgency to the task I’m about to define. Then finally, I press **Submit** and then go over to the Google Tasks web interface to see my new task recorded and ready for me to action tomorrow morning. --- # Updates to Oracle Analytics Cloud, Oracle BIEE 12c and Oracle DV Desktop Canonical: https://rittmananalytics.com/blog/2018/04/27/updates-to-oracle-analytics-cloud-oracle-biee-12c-and-oracle-dv-desktop URL: https://rittmananalytics.com/blog/2018/04/27/updates-to-oracle-analytics-cloud-oracle-biee-12c-and-oracle-dv-desktop Description: Late last year I covered one of the first releases of Oracle Analytics Cloud (OAC) on this blog covering the v3 release of OAC; since then the product has had… Author: Mark Rittman Published: 2018-04-27T01:51:36+01:00 Modified: 2018-04-27T01:51:36+01:00 --- ![Updates to Oracle Analytics Cloud, Oracle BIEE 12c and Oracle DV Desktop](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2018%2F04%2F1752277774112-b40vs8.png%3Ffit%3D800%252C683%26ssl%3D1) Late last year I covered [one of the first releases of Oracle Analytics Cloud](https://medium.com/mark-rittman/oracle-analytics-cloud-and-the-welcome-return-of-the-enterprise-bi-platform-4d0cedc4db60) (OAC) on this blog covering the v3 release of OAC; since then the product has had a major UI refresh with OAC v4 and so I thought it’d be worth covering this new release along with updates to OBIEE12c and DV Desktop in a follow-up to last year’s post. OAC’s original look and feel was based on OBIEE12c v1 that itself contained the first iteration of Oracle Data Visualisation (DV) and whilst the new modern flat UI that release introduced was an improvement on the 11g release before it since then Oracle have rapidly iterated with Oracle DV Desktop and DV Cloud Service and the UI from those products is now the primary user interface for Oracle Analytics Cloud v4. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fblog-images%2F1752277774112-b40vs8.png%3Fssl%3D1) If you’ve not used Oracle Analytics since the OAC v3 and the initial OBIEE12c versions you’ll be surprised how much the user experience has changed since then; when OBIEE12c first came out most of us still considered Answers and Dashboards to be the primary reporting UI with the BI Repository being the central carefully governed and dimensionally-modelled source of all reporting data. Since then the world has moved-on and analytics is now all about empowering the end-user through self-service, easy-to-use tools that enable you to do most tasks without having to involve the IT department or do lots of up-front data modeling and data onboarding. Oracle DV Desktop started meeting this need by introducing basic data preparation features into the product so users could upload spreadsheets and other other data and do some basic formatting and tidying before analyzing it on their desktop, and those data upload and preparation features are now available for OAC users in the v4 release. To upload a spreadsheet file into OAC you now just drop it onto the web page and you’re then given a preview and the option to upload its contents to the analytics instance. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fblog-images%2F1752277774349-g8403o.png%3Fssl%3D1) After that you can trim, split, derive and change the datatypes of data you upload so that it works in the most optimal way with the analysis features of the product, for example by turning dates in string formats not recognized by Oracle DV into proper date datatypes that can then be used in time-series analysis. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fblog-images%2F1752277774591-jmbuo8.png%3Fssl%3D1) For data transformation needs that go beyond basic formatting and datatype changes you can now build ETL-style routines that filter, aggregate, transform and add external datasets all using simple point-and-click user tools, and if you’ve licensed the new [OAC Data Lake Edition](https://blogs.oracle.com/emeapartnerbiepm/oracle-analytics-cloud-data-lake-edition-available) these transformations then extend to include model training, forecasting and sentiment analysis. In the example below I’m using OAC Data Lake Edition to forecast weight forward from a set of smart scale readings that’s then executed at the BI Server level using Python maths and statistics libraries. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fblog-images%2F1752277775381-r3bq3n.png%3Fssl%3D1) I’ll actually be talking about OAC Data Lake Edition in my presentation at ODTUG KScope’18 at Walt Disney World Florida this June, where I’ll cover some of these extended data flow features along with Oracle Big Data Cloud and Oracle Event Hub Cloud Service in my session [“BI Developer to Data Engineer with Oracle Analytics Cloud Data Lake Edition”](https://kscope18.odtug.com/page/presentations). * * * Cloud of course is great but not every Oracle Analytics customers has made the move yet, or like myself you might have a cloud instance you can spin-up as-and-when you need it but then use regular on-premises OBIEE12c in a VM for your day-to-day learning and experimentation. With Oracle’s cloud-first strategy and focus on Oracle DV as the main innovation path for analytics features this meant that my OBIEE12c instance running in a Google Compute Engine VM was starting to lag behind OAC v4 and DV Desktop in-terms of look-and-feel and all of these new features, and so I was pleased to note the other day that the 12.2.1.4.0 (aka v4) release of OBIEE12c is [now available for download on OTN](http://www.oracle.com/technetwork/middleware/bi-enterprise-edition/downloads/default-4441820.html) that includes many of the features currently available in OAC v4 and DV Desktop (but not the advanced data flow features in OAC Data Lake Edition, for example) ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fblog-images%2F1752277775658-r2vmnd.png%3Fssl%3D1) [Oracle Data Visualization Desktop](http://www.oracle.com/technetwork/middleware/oracle-data-visualization/downloads/oracle-data-visualization-desktop-2938957.html) has also been updated recently to connect to Oracle Autonomous Data Warehouse Cloud and I’ll be covering that new capability in a blog post in the near future; but more importantly for me at least…. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fblog-images%2F1752277776011-zqgq7.png%3Fssl%3D1) … it also now supports Mac OS X 10.13 High Sierra, great news for us Mac users who had to stop using DV Desktop on our laptops late last year when that most recent update to OS X came out but DV Desktop didn’t yet support it. --- # Connecting Looker to Oracle Autonomous Data Warehouse Cloud Canonical: https://rittmananalytics.com/blog/2018/04/21/connecting-looker-to-oracle-autonomous-data-warehouse-cloud URL: https://rittmananalytics.com/blog/2018/04/21/connecting-looker-to-oracle-autonomous-data-warehouse-cloud Description: Earlier in the week I wrote-up my first impressions of Oracle Autonomous Data Warehouse Cloud (ADWC) on this blog, and said at the end I’d follow with another post on… Author: Mark Rittman Published: 2018-04-21T09:19:48+01:00 Modified: 2026-07-02T00:31:07+01:00 --- ![Connecting Looker to Oracle Autonomous Data Warehouse Cloud](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F4f42b-1b31hio4ynbdlrrxweff4aq.png) Earlier in the week I wrote-up my [first impressions of Oracle Autonomous Data Warehouse Cloud](https://medium.com/mark-rittman/first-impressions-of-oracle-autonomous-data-warehouse-cloud-acebc82387f4) (ADWC) on this blog, and said at the end I’d follow with another post on how to connect ADWC to Looker, the cloud-based BI tool I usually query Google BigQuery with in my role as Analytics Product Manager at Qubit. For this initial example I uploaded a few tables of workout and other data into ADWC using SQL Developer, exporting the relevant tables out of BigQuery in CSV format and then uploading them into a user account I created for the purpose in AWDC. The table I’ll concentrate on for this example is the STRAVA\_RIDES table, each row detailing an individual cycle workout as recorded through the [Strava](https://www.strava.com/) smartphone app. ![](https://cdn-images-1.medium.com/max/800/0*zYY--GDlqw6zdcux) The first step in connecting-up Looker to ADWC, or indeed any Oracle Database, is to run some scripts that set up Oracle as Looker expects to find it. Unlike most other BI tools I’ve used with Oracle, Looker expects to connect through a specific user login (“LOOKER”) that is then granted SELECT access to any tables and views you want to report on in other schemas. This user login also needs to have some views and synonyms created to give it access to the V$ system views within Oracle that report on active sessions, and an ability to kill long-running sessions through a PL/SQL package that calls ALTER SYSTEM … KILL SESSION. The commands to run for regular Oracle databases are [detailed on the Looker website](https://docs.looker.com/setup-and-management/database-config/oracle) but you need to alter them slightly to use ADWC’s superuser account name (“ADMIN”) instead of SYS when initially connecting and when creating the LOOKER\_HASH function for symmetric aggregate handling, along with various other changes due to differences in how various objects are named in AWDS vs. regular Oracle Database. I’ve listed the commands I ran on my ADWC instance below, they should work for you but if not then check out the “Autonomous Data Warehouse Cloud for Experienced Oracle Database Users” section in [Using Autonomous Data Warehouse Cloud](https://docs.oracle.com/en/cloud/paas/autonomous-data-warehouse-cloud/user/using-oracle-autonomous-data-warehouse-cloud.pdf) that explains the differences between the new autonomous and regular Oracle Database server versions. First create the Looker account and grant the relevant roles and priviledges: connect ADMIN/<\> create user LOOKER identified by <\>; alter user LOOKER default tablespace DATA temporary tablespace TEMP account unlock; alter user LOOKER quota unlimited on DATA; alter user LOOKER default role RESOURCE; grant CREATE SESSION to LOOKER; grant UNLIMITED TABLESPACE TO LOOKER; grant CREATE TABLE TO LOOKER; grant SELECT on — ; Now create the views that Looker uses to understand what sessions are active and the SQL that’s currently being executed to provide data for looks and dashboard tiles: create or replace view LOOKER\_SQL as select SQL.SQL\_ID, SQL.SQL\_TEXT from V$SQL sql ,v$session sess where SESS.SQL\_ADDRESS = SQL.ADDRESS and SESS.USERNAME=’LOOKER’; create or replace synonym LOOKER.LOOKER\_SQL for LOOKER\_SQL; grant select ON LOOKER.LOOKER\_SQL to LOOKER; create or replace view LOOKER\_SESSION as SELECT SID, USERNAME, TYPE, STATUS, SQL\_ID, “SERIAL#”, AUDSID FROM V$SESSION WHERE USERNAME=’LOOKER’; create or replace synonym LOOKER.LOOKER\_SESSION FOR LOOKER\_SESSION; grant select ON LOOKER.LOOKER\_SESSION to LOOKER; Next, create the Oracle PL/SQL function that Looker uses as part of symmetric aggregate handling, and a function that Looker can use to “kill” runaway database queries that are taking too long to return results back to you. create or replace function LOOKER\_HASH(bytes raw, prec number) return raw as begin return(DBMS\_CRYPTO.HASH(bytes, prec)); end; create or replace synonym LOOKER.LOOKER\_HASH for LOOKER\_HASH; grant execute on LOOKER.LOOKER\_HASH to LOOKER; grant execute on ADMIN.LOOKER\_HASH to LOOKER; create or replace procedure LOOKER\_KILL\_QUERY(P\_SID in VARCHAR2, P\_SERIAL# in VARCHAR2) is CURSOR\_NAME pls\_integer default dbms\_sql.open\_cursor; IGNORE pls\_integer; begin select COUNT(\*) into IGNORE from V$SESSION where USERNAME = USER and SID = P\_SID and SERIAL# = P\_SERIAL#; if (IGNORE = 1) then dbms\_sql.parse(CURSOR\_NAME,‘alter system kill session ‘’’|| P\_SID || ‘,’ || P\_SERIAL# || ‘’’’,dbms\_sql.native); IGNORE := dbms\_sql.execute(CURSOR\_NAME); else raise\_application\_error(-20001,‘You do not own session ‘’’ || P\_SID || ‘,’ || P\_SERIAL# ||‘’’’); end if; end; create or replace synonym LOOKER.LOOKER\_KILL\_QUERY for ADMIN.LOOKER\_KILL\_QUERY; grant execute on ADMIN.LOOKER\_KILL\_QUERY to LOOKER; Next over to the Looker configuration. You’ll need to be on the Looker 5.12.12 or higher release with an instance hosted in the US to get the integration working as of the time of writing so that “ADWC” is listed as a connection type and the ADWC wallet integration works; if you’re running Looker as a hosted instance you’ll also need to speak with [support](https://discourse.looker.com/t/looker-customer-support-details/1929) to have them copy across the wallet files to the correct location on the Looker server. ![](https://cdn-images-1.medium.com/max/800/0*djjTQb9YpfspVjDl) To create the connection, enter the following details: - **Name** : Name of your connection, e.g. “rittman\_adwc” - **Dialect** : Oracle ADWC (only appears with Looker 5.12.12+) - **Host:Port** : from the TNSNAMES.ORA file in your ADWC wallet zip file - **Username** : LOOKER (as per the account setup in previous steps) - **Password** : password of LOOKER account - **Temp Databas**e : LOOKER (as per previous steps) - **Persistent Derived Tables** : checked - **Service Name** : From TNSNAMES.ORA in your ADWC wallet zip file - **Additional Database Params** : TNSNAMES.ORA SSL Server Cert DN To show the Service Name and Additional Database Params fields you first have to save the connection, then tick the “Use TNS” checkbox to reveal the fields. To find your host:port, service name and SSL Server Cert DN values first [download the wallet zip file for your ADWC instance](http://www.oracle.com/webfolder/technetwork/tutorials/obe/cloud/adwc/OBE_Provisioning_Autonomous_Data_Warehouse_Cloud_bak/provisioning_autonomous_data_warehouse_cloud_v2.html#DownloadtheCredentialsZipFile) from the ADWC Service Console, unzip the archive and then locate the details you need in the TNSNAMES.ORA file as shown below. In my case I chose to use the “medium” ADWC instance type for my connection settings. ![](https://cdn-images-1.medium.com/max/800/0*q5dCCWBi_e_ZJHED) Then, save and test your connection. The step that checks that persistent derived tables will probably fail if you try this around the time of my writing as there’s a known bug in the step that checks this feature, it’ll no doubt be fixed soon but if the rest of the checks pass you should be good. ![](https://cdn-images-1.medium.com/max/800/0*i96nd6MP1Sb1qlkU) Finally, it’s just then a case of importing your table metadata into Looker and creating explores and a model as you’d do with any other data source, like this: ![](https://cdn-images-1.medium.com/max/800/0*y9-GYtl8csOR9T7I) In this instance I’ve updated the Strava Rides LookML view to turn the relevant metric fields into measures, define a primary key for the view and remove or hide fields that aren’t relevant to my analysis, like this: ![](https://cdn-images-1.medium.com/max/800/0*7OB-2M1EWi3pq9Va) Now I can start to analyze my Strava workout data I previously uploaded to Oracle Autonomous Data Warehouse, starting with average cadence and speed along with total distance and Strava’s “suffer score” for each of my workouts: ![](https://cdn-images-1.medium.com/max/800/0*qHKXRcH_1jZwLjjW) and then looking to see how much correlation there is between distance and personal strava records being broken on my five longest rides. ![](https://cdn-images-1.medium.com/max/800/0*TmtOB7Eu-gKaPUop) In the background, Looker is sending Oracle and ADWC-specific SQL to Oracle Autonomous Data Warehouse Cloud, with the SQL tab in the Explore interface showing me the actual SQL for each query as its sent. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F4f42b-1b31hio4ynbdlrrxweff4aq.png) Should I wish to check how much of the storage and CPU capacity available for my ADWC instance is being used, I can do this from ADWC’s Service Console. ![](https://cdn-images-1.medium.com/max/800/0*HApf96skmjlR6dxS) So there you have it — Looker powered by an Oracle Autonomous Data Warehouse Cloud, and no need for an Oracle DBA to get it all running for you. --- # First Impressions of Oracle Autonomous Data Warehouse Cloud Canonical: https://rittmananalytics.com/blog/2018/04/16/2018-04-16-2018-04-16-first-impressions-of-oracle-autonomous-data-warehouse-cloud URL: https://rittmananalytics.com/blog/2018/04/16/2018-04-16-2018-04-16-first-impressions-of-oracle-autonomous-data-warehouse-cloud Description: Regular readers of this blog will know that my day-to-day work involves ingesting, storing and then analyzing petabytes of data using Google BigQuery, one of a new generation of fully-managed,… Author: Mark Rittman Published: 2018-04-16T20:24:38+00:00 Modified: 2018-04-16T20:24:38+00:00 --- ![First Impressions of Oracle Autonomous Data Warehouse Cloud](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2Fc8566-db216873-c6f3-46cf-862e-3b3fc2caf28b_448x608.png%3Fssl%3D1) Regular readers of this blog will know that my day-to-day work involves ingesting, storing and then analyzing petabytes of data using Google BigQuery, one of a new generation of fully-managed, cloud-hosted data warehouse platforms-as-a-service. Services like BigQuery take care of all the database infrastructure management and capacity planning leaving you just to define tables, load data into them and then run queries using SQL or your favourite BI tool. Oracle recently announced [Oracle Autonomous Data Warehouse Cloud](https://cloud.oracle.com/en_US/datawarehouse) (ADWC) that competes in this market and given my background in Oracle technology I was interested to see how it stacked-up compared to BigQuery and similar services from the likes of Amazon AWS and Snowflake. To do this I took advantage of Oracle’s [free $300 service credit offer](https://myservices.us.oraclecloud.com/mycloud/signup?language=en) and signed-up for an ADWC instance – note that at the time of writing the ADWC service is only available in Oracle Public Cloud’s North American datacenter region as opposed to the European ones so make sure you choose the former when creating your Oracle Cloud account. Setting up an Oracle on-premises database was something once-upon-a-time I’d spent months and thousands on course fees [studying for certification](https://groups.google.com/forum/#!topic/comp.databases.oracle.server/rn3LIOBS7Ns%5B26-50%5D)s to be able to do confidently, so it was nice to see how things had got a lot simpler with setting up an ADWC instance with the first step being to select ADWC from the Oracle Cloud Services menu, like this: ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2Fc8566-db216873-c6f3-46cf-862e-3b3fc2caf28b_448x608.png%3Fssl%3D1) Then answer a few simple questions: the name to give the instance, number of CPUs and amount of storage to set aside initially, and the password for the administrator account. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2Fc80f2-a7d17b54-315c-435a-858f-0a179adbe23d_800x358.png%3Fssl%3D1) Setup of an instance takes a few minutes, then its ready. Not bad. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2Fc5d71-f762b17d-bc5f-49e2-9576-7e0a24774623_800x345.png%3Fssl%3D1) The next big change you notice compared to Oracle’s customer-managed Database Cloud Service is the replacement of Enterprise Manager Database Express 12c and DBaaS Console with a single Service Console, accessed not through the usual SYS and SYSTEM superuser accounts but through a new superuser account called ADMIN, presumably adopting a naming standard more familiar to users of other SaaS and PaaS platforms. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2F97b0d-78922229-9d1e-452a-9344-3407fd4db682_757x376.png%3Fssl%3D1) The Service Console contains a couple of performance-tracking pages with the emphasis on read-only viewing of those stats vs. giving you controls to fine-tune database parameters or table structures, and a page where you can download a zip file containing a wallet directory with TNSNAMES.ORA and encrypted connection details to your ADWC instance. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2Fdb793-db2e2c13-6584-4a15-b1b2-742d0784db29_800x521.png%3Fssl%3D1) I seem to remember at the time of Autonomous Data Warehouse Cloud’s launch that many in the Oracle community (myself included, most probably) said that all that the “autonomous” part of Autonomous Data Warehouse Cloud did was automate the Oracle Database maintenance tasks that more recent data warehousing platforms-as-a-service handled for you automatically. Another common critique at that time was that under the covers ADWC was more scripting and use of existing “advisors” than true autonomous artificial intelligence-style database management. To be honest though, it doesn’t matter; compared to running a regular Oracle database you just fill in a short form and the platform runs without any further intervention from you, you don’t need to know how it works, job done in terms of competing with BigQuery and Redshift for ease-of-use and maintenance. Connecting to ADWC is a bit different to regular Oracle Database connections in that it’s easier — no need to specify host, port and SID/service name — but this involves downloading a ZIP file wallet and the tool you wish to connect having support for that wallet. Oracle SQL Developer in its most recent 18c version has that support making it the obvious tool to define new users, create tables and upload data into them and then run simple SQL queries against your data. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2F476cc-8f65aa0f-7f05-4448-8706-025208c80904_800x582.png%3Fssl%3D1) Oracle ADWC is based on Oracle Exadata Database Server technology and supports PL/SQL, full INSERT/UPDATE/DELETE support in SQL as well as SELECT, parallel query (enabled by default) and hybrid columnar compression. What you don’t get are indexes and materialized views as these generally aren’t considered necessary when running on Exadata hardware that’s optimized for full table scans of data arranged in compressed columnar format, and as ADWC charges per CPU per hour there’s no penalty in scanning lots of detail-level data unnecessarily as you get with BigQuery. Also missing are features such as the Oracle OLAP Option, Oracle Spatial and Graph and Oracle Text that I’m guessing might get implemented at a later date, along with management features such as Oracle Tuning Pack that are no longer needed when the autonomous features of the platform do the database tuning for you. Price-wise, although as I said a moment ago you do get charged by the amount of data you scan (and a bit for storage) with Google BigQuery, Oracle AWDC seems to be more expensive to run at least at the data volumes I’d be using as a developer initially evaluating the product. If you commit to monthly charging then ADWC [costs around $1200/cpu/month](https://cloud.oracle.com/en_US/datawarehouse/pricing) whereas Google charge $5/TB for queries with the first 1TB each month free plus a token amount for storage and streaming inserts, which meant my bill last month for BigQuery came to just $1.53 and my total bill for all Google Cloud Platform services including multiple hosted VMs and extensive usage of ther NLP and Geocoding APIs took the total up to just over $150. Clearly Google are aiming more at the startup market whereas Oracle are serving enterprise customers, but don’t make the assumption that just because AWDC is elastic it’s also super-cheap for hobbyist use, long-term. So now its a case of loading some sample data into ADWC using SQL Developer and then running a couple of test queries to see it working in action with my data. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2F71f92-ba9c13f0-6d81-44f7-8194-be7cae83a1fd_800x621.png%3Fssl%3D1) Tomorrow, I’ll show how Looker be connected using the newly announced support it has for Oracle ADWC and later on in the week do the same for Oracle Analytics Cloud and DV Desktop. --- # Timeline Charts, Derived Tables and Analytic Functions in Looker 5 Canonical: https://rittmananalytics.com/blog/2018/04/03/2018-04-03-2018-04-03-timeline-charts-derived-tables-and-analytic-functions-in-looker-5 URL: https://rittmananalytics.com/blog/2018/04/03/2018-04-03-2018-04-03-timeline-charts-derived-tables-and-analytic-functions-in-looker-5 Description: One of the data streams that comes into my personal Google BigQuery data warehouse is a record of all the Hue light switch settings and illuminance levels, Nest thermostat settings… Author: Mark Rittman Published: 2018-04-03T00:40:04+00:00 Modified: 2018-04-03T00:40:04+00:00 --- One of the data streams that comes into my personal Google BigQuery data warehouse is a record of all the Hue light switch settings and illuminance levels, Nest thermostat settings and motion sensor readings coming in via the [Samsung Smart Things hub](http://www.samsung.com/uk/smartthings/hub-f-hub-uk-v2/) under the stairs back home. For the Hue lightbulbs I get a stream 1s and 0s that initially get stored in a four-column table in BigQuery, like this: I then create a SQL view over that table that joins to a lookup table containing device types and room allocated for each of the devices in the data stream, and sets a metric type for each recording so that I can analyze all the thermostat readings together, bring together all the light switch settings and so on. Its then quite straightforward to bring this SQL view into Looker and add it to the rest of the data streams I’ve brought together there so I can see, for example, whether the kitchen lights get turned-off after everyone’s gone upstairs after breakfast or whether I should try and automate those as well as my kettle. Looker 5 introduced another way to visualize data with defined start and end events in the form of the new [Timeline Chart](https://docs.looker.com/exploring-data/visualizing-query-results/timeline-options), a feature I [tweeted about the other day](https://twitter.com/markrittman/status/975357582889095168) when I brought data on tasks from Asana into my BigQuery data warehouse via a Fivetran integration. It’d be interesting to use this type of chart to show how one particular light switch changed from on to off and then on again over a period of time, or take all the Hue lights for a given part of the house and see whether anybody turns the lights off ever … or whether they need to be linked to the motion sensors and automatically turned-off after a period of inactivity in each room. To use this type of chart I first need to create an “end date” field for each of my event records, so each timeline has a start and end date with other fields being optional for label and value. Looking at the list of events for the light switch I looked at earlier and ordering the events by time recorded you can see the information I need is there: SELECT \* FROM \`aerial-vehicle-148023.personal\_metrics.smartthings\_readings\` WHERE device = ‘kitchen seating area spot 1’ AND raw\_value IN (‘off’,’on’) ORDER BY date\_time What this calls for is the LEAD BigQuery Standard SQL [analytic function](https://cloud.google.com/bigquery/docs/reference/standard-sql/functions-and-operators#analytic-function-syntax) that returns a value based on a subsequent row to the one we’re working with, and I can combine that with the [TIMESTAMP\_DIFF](https://cloud.google.com/bigquery/docs/reference/standard-sql/functions-and-operators#timestamp_diff) function to compute the time between the start and end event dates as that would be useful to know too. The SQL expressions for the two new fields would therefore be: LEAD(date\_time, 1) OVER (PARTITION BY device, metric ORDER BY date\_time) end\_date\_time, timestamp\_DIFF(LEAD(date\_time, 1) OVER (PARTITION BY device, metric ORDER BY date\_time),date\_time,second) as value\_duration\_seconds I could add these column expressions into the SQL view definition in BigQuery and order the rows by device, metric and timestamp, like this: SELECT date\_time as date\_time, device as device, metric, value, raw\_value, LEAD(date\_time, 1) OVER (PARTITION BY device, metric ORDER BY date\_time) as end\_date\_time, timestamp\_DIFF(LEAD(date\_time, 1) OVER (PARTITION BY device, metric ORDER BY date\_time),date\_time,second) as value\_duration\_seconds from \`aerial-vehicle-148023.personal\_metrics.smartthings\_readings\` where device = ‘kitchen seating area spot 1’ and metric = ‘switch’ and raw\_value in (‘on’,’off’) order by 2,3,1 Executing that SQL shows the logic is working, but I’d then have to maintain that view within BigQuery and that might not be the most convenient place to add new code. Instead, I could just go into Looker and create a new view there based on a derived table SQL expression and do the work there. Right now my LookML model looks like the excerpt below, where you can see the sensor readings view joined into the rest of the explore so all my data can be analyzed together. connection: “rittman\_bigquery” include: “\*.view.lkml” # include all views in this project include: “\*.dashboard.lookml” # include all dashboards in this project explore: date\_dim { case\_sensitive: no label: “Data Warehouse” join: fluentd\_transactions { type: left\_outer sql\_on: ${date\_dim.date\_minute5} = ${fluentd\_transactions.date\_minute5} ;; relationship: many\_to\_many } join: smartthings\_readings { type: left\_outer sql\_on: ${date\_dim.date\_minute5} = ${smartthings\_readings.date\_minute5} ;; relationship: one\_to\_many } join: fluentd\_uber\_rides { type: left\_outer sql\_on: ${date\_dim.date\_minute5} = ${fluentd\_uber\_rides.date\_minute5} ;; relationship: many\_to\_many } } I now create a new LookML view that uses a derived table SQL query as the view definition rather than simply referencing an existing BigQuery table. Note how I’ve used the same view\_label as the main LookML view containing my event data, so that the dimensions and metric I define here appear alongside all the other smart device fields the same explore view. view: device\_event\_end\_and\_timespan { view\_label: “6 – Smart Devices” derived\_table: { sql: SELECT date\_time as date\_time, device as device, metric as metric, LEAD(date\_time, 1) OVER (PARTITION BY device, metric ORDER BY date\_time) as end\_date\_time, timestamp\_DIFF(LEAD(date\_time, 1) OVER (PARTITION BY device, metric ORDER BY date\_time),date\_time,second) as value\_duration\_seconds from \`aerial-vehicle-148023.personal\_metrics.smartthings\_readings\` order by 2,3,1;; } dimension: date\_time { type: date\_time hidden: yes sql: ${TABLE}.date\_time ;; } dimension: device { type: string hidden: yes sql: ${TABLE}.device ;; } dimension: metric { type: string hidden: yes sql: ${TABLE}.metric ;; } dimension\_group: end\_date\_time { group\_label: “End Date” label: “End” type: time timeframes: \[ raw, time, hour, hour3, hour4, hour6, hour12, hour\_of\_day, time\_of\_day, minute, minute5, minute10, minute15, minute30, day\_of\_week, day\_of\_month, day\_of\_year, date, week, week\_of\_year, month, month\_name, month\_num, quarter, year \] sql: ${TABLE}.end\_date\_time ;; } dimension: pk { primary\_key: yes hidden: yes type: string sql: concat(cast(${TABLE}.start\_date\_time as string), ${TABLE}.device) ;; } measure: value\_duration\_seconds { type: average sql: ${TABLE}.value\_duration\_seconds ;; } } Then I join this derived table view back into the explore within my LookML model, as below: connection: “rittman\_bigquery” include: “\*.view.lkml” # include all views in this project include: “\*.dashboard.lookml” # include all dashboards in this project explore: date\_dim { case\_sensitive: no label: “Data Warehouse” join: fluentd\_transactions { type: left\_outer sql\_on: ${date\_dim.date\_minute5} = ${fluentd\_transactions.date\_minute5} ;; relationship: many\_to\_many } join: smartthings\_readings { type: left\_outer sql\_on: ${date\_dim.date\_minute5} = ${smartthings\_readings.date\_minute5} ;; relationship: one\_to\_many } join: device\_event\_end\_and\_timespan { type: left\_outer sql\_on: ${smartthings\_readings.date\_time} = ${device\_event\_end\_and\_timespan.date\_time} and ${smartthings\_readings.device} = ${device\_event\_end\_and\_timespan.device} and ${smartthings\_readings.metric} = ${device\_event\_end\_and\_timespan.metric};; relationship: one\_to\_one } join: fluentd\_uber\_rides { type: left\_outer sql\_on: ${date\_dim.date\_minute5} = ${fluentd\_uber\_rides.date\_minute5} ;; relationship: many\_to\_many } and now I can create timeline charts that show me which lights were on over a particular period of time, like this: or more importantly, work out why the bathroom never seems to be free when you’ve got two teenage kids in the house. --- # The Drill to Detail Podcast : 50th Episode Special and Top 10 Episodes by Download Canonical: https://rittmananalytics.com/blog/2018/03/05/the-drill-to-detail-podcast-50th-episode-special-and-top-10-episodes-by-download URL: https://rittmananalytics.com/blog/2018/03/05/the-drill-to-detail-podcast-50th-episode-special-and-top-10-episodes-by-download Description: I started the Drill to Detail Podcast series back in 2016 in order to talk about some of the innovations coming out of the big data analytics space with the… Author: Mark Rittman Published: 2018-03-05T01:05:43+01:00 Modified: 2018-03-05T01:05:43+01:00 --- ![The Drill to Detail Podcast : 50th Episode Special and Top 10 Episodes by Download](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2018%2F03%2F1752277778276-qs3euu.png%3Ffit%3D234%252C232%26ssl%3D1) ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fblog-images%2F1752277778276-qs3euu.png%3Fssl%3D1) I started the Drill to Detail Podcast series back in 2016 in order to talk about some of the innovations coming out of the big data analytics space with the product teams, industry analysts and consultants working together to deliver new-world, cloud-hosted big data analytics platforms. A year-and-a-half later we’re at Episode 50 and to mark the occasion Stewart Bryson, our first guest on the show back at Episode 1 and a regular on the show since, comes back on and is joined by Alex Gorbachev to talk about Google Cloud Platform, Looker, FiveTran and SnowflakeDB, with the [episode now available for downloa](https://www.drilltodetail.com/podcast/2018/2/27/drill-to-detail-ep50-agile-bi-karl-marx-and-our-man-from-moscow-with-special-guests-stewart-bryson-and-alex-gorbachev)d along with all 49 other episodes on the [Drill to Detail Podcast website](https://www.drilltodetail.com/) … … and as we’re now at the 50th episode of the show, here’s the Top 10 episodes by download statistics as of January 2018 — thanks to everyone who’s come on the show and everyone who’s listened to the show via iTunes and other podcast providers, and congratulations to Maxime Beauchemin: 1. [Drill to Detail Ep.26 ‘Airflow, Superset & The Rise of the Data Engineer’ with Special Guest Maxime Beauchemin](https://www.drilltodetail.com/podcast/2017/5/15/drill-to-detail-ep26-airflow-superset-the-rise-of-the-data-engineer-with-special-guest-maxime-beauchemin) 2. [Drill to Detail Ep.21 ‘The Fall, and Rise, of Microsoft BI’ With Special Guest Chris Webb](https://www.drilltodetail.com/podcast/2017/3/14/drill-to-detail-ep21-the-fall-and-rise-of-microsoft-bi-with-special-guest-chris-webb) 3. [Drill to Detail Ep.31 ‘Dremel, Druid and Data Modeling on Google BigQuery’ With Special Guest Dan McClary](https://www.drilltodetail.com/podcast/2017/6/19/drill-to-detail-ep31-dremel-druid-and-data-modeling-on-google-bigquery-with-special-guest-dan-mcclary) 4. [Drill to Detail Ep.34 ‘Qlik, MSAS and the Strategy of Data Analytics’ with Special Guest Donald Farmer](https://www.drilltodetail.com/podcast/2017/7/12/drill-to-detail-ep34-qlik-msas-and-the-strategy-of-data-analytics-with-special-guest-donald-farmer) 5. [Drill to Detail Ep.48 ‘Mondrian OLAP, Apache Calcite and Database Dis-Aggregation’ With Special Guest Julian Hyde](https://www.drilltodetail.com/podcast/2018/1/23/drill-to-detail-ep48-mondrian-olap-apache-calcite-and-database-dis-aggregation-with-special-guest-julian-hyde) 6. [Drill to Detail Ep.25 ‘SAP BusinessObjects Analytics, AI and Digital Transformation’ With Special Guest Timo Elliott](https://www.drilltodetail.com/podcast/2017/5/1/drill-to-detail-ep25-sap-businessobjects-analytics-and-digital-transformation-with-special-guest-timo-elliott) 7. [Drill to Detail Ep.27 ‘Apache Kafka, Streaming Data Integration and Schema Registry’ with Special Guest Gwen Shapira](https://mark-rittman-lzl4.squarespace.com/podcast/2017/5/22/drill-to-detail-ep27-apache-kafka-streaming-data-integration-and-schema-registry-with-special-guest-gwen-shapira) 8. [Drill to Detail Ep. 5 ‘SnowflakeDB, and is Data Modeling Dead?’ with Special Guest Kent Graziano](https://mark-rittman-lzl4.squarespace.com/podcast/2016/9/14/yffvlkgzk1099zpz944v4ttfd9njp1) 9. [Drill to Detail Ep.23 ‘Looker, BigQuery and Analytics on Big Data’ With Special Guest Daniel Mintz](https://www.drilltodetail.com/podcast/2017/3/28/drill-to-detail-ep23-looker-bigquery-and-analytics-on-big-data-with-special-guest-daniel-mintz-1) 10. [Drill to Detail Ep.6 ‘Data Capital and the Economics of Big Data’ with Special Guest Paul Sonderegger](https://www.drilltodetail.com/podcast/2016/10/23/rsmn3fbumx1p7zew7kbopykm5x6qa2) --- # Monzo, BigQuery, Looker and “Fintech”: The Other London Tech Startup Scene Canonical: https://rittmananalytics.com/blog/2018/02/05/monzo-bigquery-looker-and-fintech-the-other-london-tech-startup-scene URL: https://rittmananalytics.com/blog/2018/02/05/monzo-bigquery-looker-and-fintech-the-other-london-tech-startup-scene Description: I’ve written a number of blogs over the past few months about bringing my fitness, smart devices and other digital data sources into Google BigQuery and Looker to do interesting… Author: Mark Rittman Published: 2018-02-05T22:32:14+00:00 Modified: 2026-07-02T00:31:13+01:00 --- ![Monzo, BigQuery, Looker and “Fintech”: The Other London Tech Startup Scene](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F1de4c-1vtkev1n9y6bgd9b5pnkwrw.png) I’ve written a number of blogs over the past few months about [bringing my fitness, smart devices and other digital data sources into Google BigQuery and Looker](https://medium.com/mark-rittman/what-bi-development-looks-like-with-bigquery-google-cloud-apis-looker-and-fluent-courtesy-of-the-f9e8c23ba0c6) to do interesting analysis with the combined dataset, whilst also keep my tech skills up to date and relevant for my day job as Product Manager for Analytics at Qubit, a London tech startup in the hot new [MarTech](https://martechtoday.com/qubit-launches-instagram-like-ai-powered-product-news-feed-mobile-205333) (“Marketing Technology”) space. [At Qubit we also use Google BigQuery and Looker](https://www.qubit.com/webinars/analytics-for-personalization/) to enable our customers to run fast, interactive queries against petabytes of consumer behavioural data whilst BigQuery abstracts away the hassles in running a production data warehouse, whilst Looker abstracts away the complexities of SQL, joining datasets together and trying to visualize it all in D3 or RStudio. One big gap in the data I’d managed to collect up until recently was anything to do with what I actually spent each day; whilst my current bank along with most others provides a means to manually download statement data in Excel or CSV format, or supports commercial services that load that data into your personal finance software for a monthly fee, I wanted to access my data in the same way that I accessed my [health](https://www.humanapi.co/), [timekeeping](https://www.rescuetime.com/developers) and [cycling workout data](https://strava.github.io/api/) … through a REST API, cURL and a cron job that loads new data into BigQuery on the hour every day. So that’s why I smiled when I heard about [Monzo](https://monzo.com/), another tech startup just a mile or so from my office in London that’s disrupting another industry: consumer banking, with Monzo, Starling and other new app-only banks taking advantage of new regulations such as the EU’s new [Payment Services Directive (PSD2)](https://www.cnbc.com/2017/12/25/psd2-europes-banks-brace-for-new-eu-data-sharing-rules.html) that open up existing bank accounts to authorised third-parties looking to provide a better user experience (UX) over your existing bank arrangements; or as in Monzo’s case, actually obtaining a banking license and offering a complete end-to-end service that focuses on solving customer problems rather than you providing a cheap source of money to lend for the bank. Along with other technology-led startups working in the financial services space Monzo are the best example yet of a compelling new financial services company with a strong appeal to consumers more familiar today with [Facebook](https://www.theguardian.com/technology/2017/dec/17/monzo-facebook-of-banking) than chequebooks. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F1de4c-1vtkev1n9y6bgd9b5pnkwrw.png) Image Courtesy of Medici, [“Awe-Inspiring London FinTech Industry Is Firing on All Cylinders”](https://gomedici.com/awe-inspiring-london-fintech-industry-is-firing-on-all-cylinders/) Me, I was more interested in the fact that [Monzo also use Looker and Google BigQuery](https://monzo.com/blog/2016/11/30/laying-the-foundation-for-a-data-team/) to run their analytics infrastructure, and that they also offer a [developer-orientated REST API](https://developers.monzo.com/) that provides exactly what I’ve been looking for to enable myself and increasingly [others in the developer community](https://github.com/rdingwall/awesome-monzo) to download, analyze and visualize their spending data and start to build applications that [when the full Monzo API comes out](https://monzo.com/blog/2018/01/12/api-update/) will revolutionize how we do our banking in the coming years. * * * To start analyzing my banking data through Monzo I first needed to do a historic batch download of all my historic transactions and top-ups and put that into a BigQuery table, and then setup a trigger on Monzo’s systems that sends across all the subsequent transactions as they happen to keep my data in-sync with Monzo’s record of my bank account. To set up the initial transaction history transfer I first registered at [https://developers.monzo.com/](https://developers.monzo.com/) and then generated an access token for my original pre-paid Monzo card and the current account card that recently replaced it. Note that this access token only lasts for a short time so you can’t generate it just once and use it in a script forever, and more importantly the developer API is meant just for personal use and can’t be used to access other people’s accounts, so you can’t build one of the new apps made possible by the UK’s new open banking regulations just yet (but this is coming soon through their [interim AIS API now in testing](https://monzo.com/blog/2018/01/12/api-update/) with a small number of third-party developers) ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F2cae6-1yev-4ldpb6hov1yrp1t4bg.png) To find out the account IDs for your Monzo accounts either use the API playground web app as I’ve done in the screenshot above, or use the Developer API now for the first time to get them yourself, via the cURL tool: curl -G “[https://api.monzo.com/accounts](https://api.monzo.com/accounts)" -H “Authorization: Bearer <\>" ./accounts.json and then retrieve your account IDs from the JSON output through a tool such as jsonpp (brew install jsonpp if you’re on a Mac, similar tools for other OSs) { "accounts": \[ { "id": "acc\_<\>" "created": "2017-04-06T07:07:15.175Z", "description": "Mark Rittman", "type": "uk\_prepaid" }, { "id": "acc\_<\>" "created": "2017-11-28T08:35:28.323Z", "description": "Mark Robert Rittman", "account\_number": "<", "sort\_code": "<\>", "type": "uk\_retail" } \] } Then you can retrieve all your account transactions with another cURL request, like this one for my current account: curl -G "[https://api.monzo.com/transactions](https://api.monzo.com/transactions)" -d account\_id=acc\_<\> -H "Authorization: Bearer <\>" \> /transactions.json and then you can use tools such as “sed” (to strip out the transactions: \[\]) array from the json output) and one of the many json2csv converters out there (for example [this one](https://github.com/zemirco/json2csv) that has npm as a prerequisite, something I’m guessing most people attempting this sort of project will have already, if not then first [install npm via brew](https://changelog.com/posts/install-node-js-with-homebrew-on-os-x) and you’re ready) sed -i -e 's/{"transactions"://g' ./transactions.json sed -i -e 's/\]}/\]/g' ./transactions.json brew install json2csv json2csv -i ./transactions.json -f created,amount ,description,currency,category,include\_in\_spending,<\> -o transactions.csv There are many more fields available to extract from the JSON documented downloaded via the developer API depending on whether you’re accessing a Monzo prepaid card or current account; the developer docs on the Monzo site [go through some of them](https://monzo.com/docs/#retrieve-transaction) but you’re best going through a handful of transactions to see what’s available and then create your own custom -f extract list. Then as the final step I use the bq command-line tool to load the data in the CSV file output in the previous step into a BigQuery table, like this: bq load --source\_format=CSV --replace=true --skip\_leading\_rows=1 --autodetect dw\_staging.monzo\_transactions\_stg ./transactions.csv and then go into the BigQuery Web UI to check my transaction data has come in as expected. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2Fc3f68-1zv8k-fqv_uoinvud-nemxw.png) In-practice for my analysis exercise I brought in a lot more fields including merchant name and map X/Y location, [Foursquare](https://venturebeat.com/2013/02/26/foursquare-visa-mastercard/) metadata and other fields that are only available for the current account product, so check through the JSON elements available for your particular Monzo product and extract the ones that most interest you. * * * So what about capturing transactions after this date, how do we do that without some complicated process that downloads all the transactions again and then applies just the ones that are new to the BigQuery table? The way I bring in all my other app, smart device and other event data into BigQuery is to either post them directly to a fluentd server running on Google Compute Engine or use services such as IFTTT and [Zapier](https://zapier.com/blog/how-use-zapier-webhooks/) that interface with my apps and then post to fluentd using outgoing [webhook services](https://ifttt.com/maker_webhooks), or their “Maker Channel” as IFTTT used to call their service when I created for a [blog post back in 2016](https://medium.com/mark-rittman/what-bi-development-looks-like-with-bigquery-google-cloud-apis-looker-and-fluent-courtesy-of-the-f9e8c23ba0c6). ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2Fee7e2-12qn-t4s0fyumbfweykzqka.png) The Monzo Developer API also comes with a [webhook feature](https://github.com/monzo/docs/blob/master/source/includes/_webhooks.md) that lets you register a URL, for example the one use when posting all my other event data to fluentd, with all subsequent transaction data then being posted to that server URL and then onto the various BigQuery tables that store my data for analysis. In the case of the Monzo transactions I added an intermediate step and had the Monzo webhook post to a Zapier “zap” that made it easier to decode the nested JSON elements Monzo was sending over that held merchant and other payment processor-related information. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2Fb0394-1nxmvhqpb0if06k1uu30d8w.png) Zapier then sends over just those elements of the transaction I’m interested in to the fluentd server which then streams them into BigQuery using the [fluentd BigQuery plugin](https://github.com/kaizenplatform/fluent-plugin-bigquery). From there I can then write SQL to analyze this data in BigQuery’s web UI, remembering to divide the transaction amount by 100 as they’re all stored as pence in integer datatypes within the Monzo system, and to remove any top-ups, declines and and other non-spend transactions from the results by filtering on the “include\_in\_spending” column to remove all records with “false” as the predicate value. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F504a5-1wu4refht4iqzntfrjqa6rq.png) * * * Analyzing the data in a tool such as [Looker](https://looker.com/) makes things much more interesting as I can start to make use of the locational metadata for merchants that Monzo provide for current account transactions, showing me for example where I took out cash from ATMs on the way back home from the pub and then forgot I’d done so the morning after. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F7710b-1rgchdrwaxyipjwszj85tea.png) Or I can see how my spend on essentials like entertainment and eating out rises in the summer, especially when I appear to do away with non-essentials like food and other groceries. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F464de-1klnu46hxjvlprwwor5jova.png) I can see where it was I spend the most money when [out on cycling trips](https://www.strava.com/athletes/13170184?hl=en-GB) in the summer, mostly in pubs around the New Forest and up near Bath, Frome and Warminster (and how I wish winter would end and I can get back out on my bike again). ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2Fecae8-1bnh6t3rbwnx_1bq-le-z0w.png) I can even work out how my choice of restaurant and entertainment changed from lots of spend on music in August through to lots of eating out in October. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F7c82f-10iuhbanju8csmr_qchxguw.png) And more importantly, I can see I’m spending far too much money on coffees and breakfast on the way into work each day, but that’s the downside I guess of working just off Covent Garden in what must be one of the nicest work locations I’ve ever had the privilege to base myself out of. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F8169c-1guyfhkumhb1u9wcttbyosg.png) The beauty of Looker is that I can define a data model in LookML that takes table structures and SQL expressions and turns them into more user-friendly measure and dimension definitions, like this: ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2Fc8465-11_xma0ntsyemfrfp8e9uga.png) These LookML “views” can then be joined to other views that share a common join key — in my case, the truncated “date” part of a date and time timestamp field — that then makes it possible for me to analyze spend data in the context of where I was on that day and that time; whether I was out on the bike and if so whether I’d had a good workout or not so good; what the weather was like on the day and what I’d been chatting about on Slack at the time; whether the spend was for an Uber ride and if so where to. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2026%2F07%2F8bd84-19ghcmw-c7j9m2apf9y4x2a.png) Analysis that shows me, now I’m finally able to join my financial and activity dots together for the first time, where all my money’s gone since the nights got darker and I’m clearly in the pub rather than out on my bike. Lucky spring is around the corner or I’ll be applying for [Monzo’s new overdraft service](https://monzo.com/blog/2017/08/24/overdraft-preview/) that’s just gone into review rather than registering new Strava personal records next month. --- # Oracle Analytics Cloud … and the (Welcome) Return of the Enterprise BI Platform Canonical: https://rittmananalytics.com/blog/2017/12/31/2017-12-31-2017-12-31-oracle-analytics-cloud-and-the-welcome-return-of-the-enterprise-bi-platform URL: https://rittmananalytics.com/blog/2017/12/31/2017-12-31-2017-12-31-oracle-analytics-cloud-and-the-welcome-return-of-the-enterprise-bi-platform Description: Oracle Analytics Cloud, the “full” implementation of Oracle’s business analytics platform tools in Oracle Public Cloud, was released back in May 2017 and I covered its basic product proposition in… Author: Mark Rittman Published: 2017-12-31T15:19:13+00:00 Modified: 2017-12-31T15:19:13+00:00 --- ![Oracle Analytics Cloud … and the (Welcome) Return of the Enterprise BI Platform](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2Fdec13-058e60bc-6d5d-45de-b46b-791f6767c96e_768x532.png%3Fssl%3D1) [Oracle Analytics Cloud](https://cloud.oracle.com/en_US/oac), the “full” implementation of Oracle’s business analytics platform tools in Oracle Public Cloud, was released back in May 2017 and I covered its basic product proposition in a [Drill to Detail Podcast episode](https://www.drilltodetail.com/podcast/2017/2/8/drill-to-detail-ep18-oracles-cloud-analytics-and-data-visualization-strategy-with-special-guest-vasu-murthy) with Oracle’s Vasu Murthy a few weeks just before then. Up until now I’d not really had a chance to properly review Oracle Analytics Cloud as the products I work with right now are mainly based around Google’s Cloud Platform and the Looker BI tool, but Oracle kindly made available some cloud promotion credits through their [Oracle ACE Director community program](http://www.oracle.com/technetwork/community/oracle-ace/become-an-ace/index.html#acelevels) and so I gave it as well as a few other supporting cloud services a spin over the Christmas break. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2Fdec13-058e60bc-6d5d-45de-b46b-791f6767c96e_768x532.png%3Fssl%3D1) In the end it was only really a high-level evaluation but with my previous experience with Oracle’s BI Cloud Service, Data Visualization Desktop and Cloud Services and the on-premises versions of their BI Enterprise Edition product line I was able to get up and running fairly quickly. More interestingly for me was trying to work out who the target market is for Oracle Analytics Cloud and what particular use-cases it tries to address; why Oracle released Oracle Analytics Cloud when they already have [Oracle BI Cloud Service](https://cloud.oracle.com/business_intelligence) and how they differentiate the two products and the two markets they presumably serve; and to try and speculate on why Oracle made the choices it made when coming-up with Oracle Analytics Cloud given the competition it faces in the market. * * * For anyone not so familiar with Oracle’s business analytics products you’ve basically got a classic end-to-end enterprise BI platform, Oracle Business Intelligence 12c, that covers everything from reporting tools through to business metadata modeling to application and database server integration that typically sold to the IT function within corporate and government customers and was recently joined by Oracle Data Visualization that turned this all on its head and sold to the sorts of buyers who were now buying Tableau on their company credit cards and bypassing the IT department so that they could actually get something done. Oracle’s first forays into moving all this into the cloud were focused again on the needs of these non-technical business buyers starting with Oracle Business Intelligence Cloud Service (“BICS”), a departmental cut-down version of Oracle Business Intelligence that I [covered in an article for Oracle Magazine](http://www.oracle.com/technetwork/issue-archive/2015/15-may/o35ba-2541604.html) when the product first came out. Over time [BICS was extended to include data visualization capabilities](http://www.oracle.com/technetwork/issue-archive/2015/15-sep/o55ba-2692803.html) that were being introduced with the Data Visualization Option for the new Oracle Business Intelligence 12c release and then just those visualization features were packaged up into another end-user focused cloud service, [Oracle Data Visualization Cloud Service](https://cloud.oracle.com/data_visualization), that became available around two years ago. One thing you couldn’t do with any of those cloud tools though was migrate your on-premises Oracle BI deployments into Oracle’s Public Cloud, as BICS made you use a cut-down version of the Oracle Database along with a much-simplified data modeling tool to store and organize your data for reporting, but around the same time as Data Visualization Cloud Service came out it started to become possible to [“lift-and-shift” your full enterprise business metadata models into BICS](https://www.slideshare.net/rittmanmead/deploying-full-oracle-bi-platforms-to-oracle-cloud-oow2015) and run them against full Oracle Database instance running in the cloud or on-premises databases using a special secure connector between Oracle’s cloud data centres and the one running your Oracle database servers. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2F8ba33-ba0e6572-004c-43c8-a9a2-bcb2366778b0_800x523.png%3Fssl%3D1) “Lifting and Shifting” an on-premises RPD into Oracle BI Cloud Service What some customers still wanted though was more than this; BICS, DVCS and all the other BI products Oracle offered were fully-managed services, not unlike Looker and Google’s BigQuery distributed database service that I use day-to-day in my current product management role at Qubit. What these customers wanted was full on-premise Oracle Business Intelligence running in Oracle Public Cloud that they could then connect into securely and manage the configuration settings to suit just their particular needs, and choose when to apply patches and when to run backups to suit their standard operating model. What they also wanted was full unfettered access to Oracle’s BI Server metadata layer, so they could not only upload their on-premises business metadata models but then extend and update them and at the same time incorporate elements of functionality from Oracle’s Essbase Server that had also recently made its own transition into Oracle Public Cloud. Crucially, these customers still wanted to pay for all of these software costs monthly out of OpEx but also wanted the license metric to move back to per-processor rather than the named-used basis BICS and DVCS used, so they could then roll-out cloud analytics to everyone in the organization rather than just a few users and also spin-up test, development and pre-production environments if they’d licensed enough processor capacity to run them all. This, then, was the set of requirements Oracle Analytics Cloud was put together to meet, and so I took a look at what v1 of this new product looked like over the time between Christmas and New Year. * * * Installing Oracle Analytics Cloud is more like installing Tableau Server, or one of the other pre-rolled BI server software VMs you find on services such as [AWS Marketplace](https://aws.amazon.com/mp/), than than getting a new account on BICS or one of the other Oracle fully-managed service running on Oracle Public Cloud. You first have to set up some object storage space using Oracle Cloud Storage to hold the various configuration and log files OAC will require, then setup a database instance using Oracle Database Cloud Service to hold the RCU schema that OAC will require. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2Fdeea5-07e41ca6-8642-4532-86e5-da86b0c9bbda_800x557.png%3Fssl%3D1) Creating the Oracle Analytics Cloud service Oh, and you almost definitely need some prior knowledge of how to setup and configure Oracle software, and the previous on-premises version of Oracle Business Intelligence, before any of this makes sense; Oracle Analytics Cloud is definitely aimed at the IT department rather than casual business users, and moreover those IT departments already invested in Oracle’s analytics, database and cloud technologies for whom this will seem very familiar and in-fact will seem very reassuring — this is Oracle’s full on-premise functionality ported up into the cloud, not some cut-down managed-for-you version that hides all the controls and is impossible to manage via your own DevOps scripting. Truth be told, I found the installation process just baffling when I first ran through it; being more used to services such as Looker and Google’s BigQuery, Cloud DataPrep and Google Data Studio products these days being presented with software where you *actually need to understand the installation process and what it’s doing*, and that process requires mandatory knowledge of other current and historic products relating to the one I was installing, seemed crazy in-comparison to the startup-friendly SaaS products I’m more familiar with these days. But every organization isn’t a startup, and every user of an installer utility isn’t a non-technical business user; complex enterprise software is complex because enterprises, and large government customers have gone beyond simple customer success departments, growth hackers and biz ops teams to instead needing to support operations across multiple countries, integrate with legacy systems you’ve never heard of and hopefully never will, and do everything according to strict sets of rules such as [Sarbanes-Oxley](http://www.dbta.com/Columns/SQL-Server-Drill-Down/HIPAA-DBAs-Data-and-the-Law-121155.aspx) and [HIPAA](http://www.dbta.com/Columns/SQL-Server-Drill-Down/HIPAA-DBAs-Data-and-the-Law-121155.aspx) that land you in jail if you don’t comply with them. These types of organization have very strict rules and procedures around how software is hosted, accessed and managed and vendors like Oracle know these customers’ needs well, who would in most cases prefer a complex but controllable installation process over one that hides all the details but potentially leaves them exposed to regulatory issues thereafter, meaning they can’t ever sign-off the provisioning process as complete and fully-compliant. And one of the first lessons I learnt in product management is that whilst engineers and product managers prefer packages that give end-users all available functionality for one simple price, enterprise salespeople much prefer *options*. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2F20225-eef6b4c1-156e-46d3-a538-95ba85bd4935_627x314.png%3Fssl%3D1) PMs like packages, enterprise sales prefer options With packages sold to enterprises with a single list price advertised — say, $1m all-in for the right to run everything on a single high-powered server — what happens in reality is that one or two customers pay that price, but the vast majority get some sort of discount so that everybody pays something and no sales are lost just because the customer couldn’t afford the list price. What works far better when selling enterprise deals is when everything beyond a basic core set of functionality becomes an option and customers can then align what they want with what they can afford without the vendor giving everything away each time and massively-discounting deals to make them affordable. Hence Oracle Analytics Cloud having both Standard and Enterprise Editions, and the Standard Edition having two variants based around just data visualization or just Essbase, and there being an upcoming Data Lake Edition that will include other net-new functionality, and presumably some time in the future other functionality being added as additional options that will need further license spend by the customer before becoming available to their end-users. Just remember when you’re sitting in the engineering department mocking the salespeople that extracting the right price from the right customers whilst everyone feeling they’ve got a good deal is what b2b selling is all about, which is why those salespeople are about the only people who are paid more than the engineers in most startups. * * * So what does Oracle Analytics Cloud look like once you’ve uploaded or pointed it towards some data, and started to visualize and analyze it using the various tools available to you in the platform? Well it’s effectively Oracle Business Intelligence 12c with the Data Visualization Option, at least in the configuration I’d chosen; “Self-Service Data Visualization, Preparation and Smart Discovery” along with “Enterprise Data Models” which translates to Visual Analyzer plus Answers, Dashboards and a full RPD in old on-premise terms. There’s the new home page inherited from Visual Analyzer and a new console that allows you to perform admin functions, define and work with data sources and download the BI Administration tool if the online modeler is too simplistic for your project. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2F02665-b1416a7f-2eb0-45f6-85e8-b8927d8f2b7e_800x543.png%3Fssl%3D1) Oracle Analytics Cloud Console and Home Page I was able to upload and then analyze some of my cycling stats from Strava, working out that Saturdays in September were my best time for logging the miles and working off some calories in the last twelve months. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2F79b33-35e913f0-c439-4846-b8ae-52bce2e19c1f_800x557.png%3Fssl%3D1) Visual Analyzer within Oracle Analytics Cloud I was also pleased to see that the classic Answers interface was also still available along with the original home page, recent and popular content and task menus. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2Ff2ca9-a115e51c-e503-4299-96d5-b6b7b21ea643_800x530.png%3Fssl%3D1) Classic Answers interface from OBIEE12c within Oracle Analytics Cloud And this gets to what the central appeal of Oracle Analytics Cloud really is; it’s full Oracle Business Intelligence 12c, Oracle Data Visualization and Oracle Essbase with all the configuration options, IT appeal and product capabilities that were built-up over the years based on the requirements of large-scale, sophisticated and demanding enterprise and government customers. It’s not [Superset](https://medium.com/airbnb-engineering/superset-scaling-data-access-and-visual-insights-at-airbnb-3ce3e9b88a7f) or Looker, it makes no sense to anyone who’s not already invested in the Oracle analytics and database ecosystems and it can meet just about any analytics requirement, and then some … and now it all runs natively in the cloud. Sadly my trial account and promotional credits ran out after a couple of weeks but if Oracle are ever kind enough to extend that access sometime in the New Year I’d be glad to roll-up my sleeves, [dust-down my old book on how it all works](https://www.amazon.co.uk/Oracle-Business-Intelligence-Developers-Guide/dp/0071798749/ref=la_B00B8YEQMU_1_1?s=books&ie=UTF8&qid=1514732055&sr=1-1) and give this new cloud incarnation of the product that defined my career for the best part of ten years a further look. --- # UKOUG Tech’17 and the Incredible World of eCommerce Analytics, Machine Learning and 1:1 Marketing Canonical: https://rittmananalytics.com/blog/2017/12/03/2017-12-03-2017-12-03-ukoug-tech17-and-the-incredible-world-of-ecommerce-analytics-machine-learning-and-11-marketing URL: https://rittmananalytics.com/blog/2017/12/03/2017-12-03-2017-12-03-ukoug-tech17-and-the-incredible-world-of-ecommerce-analytics-machine-learning-and-11-marketing Description: Although my main focus this past year has been in product management and the vendors and technologies that London’s FinTech and MarTech startups tend to use, my professional roots are… Author: Mark Rittman Published: 2017-12-03T22:40:32+00:00 Modified: 2017-12-03T22:40:32+00:00 --- ![UKOUG Tech’17 and the Incredible World of eCommerce Analytics, Machine Learning and 1:1 Marketing](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2F60d67-34fb8a2d-073e-494b-b5bf-d73ae3d1dbf8_800x468.png%3Fssl%3D1) Although my main focus this past year has been in product management and the vendors and technologies that [London’s FinTech](https://www.ft.com/content/112c6932-bf37-11e3-a4af-00144feabdc0) and [MarTech startups](https://www.techworld.com/picture-gallery/startups/top-12-marketing-tech-martech-startups-in-uk-3639921/) tend to use, my professional roots are in the Oracle world and I still maintain a strong interest in what’s new and innovative coming out of Redwood Shores such as the new [Analytic Views feature introduced with Oracle Database 12c](https://medium.com/mark-rittman/analytic-views-oracle-database-12-2-and-oracles-reboot-of-olap-for-in-memory-analysis-5d7eedc02b33) I covered on the blog back in May this year. I’m particularly pleased therefore to be travelling up to Birmingham this week to attend the [UK Oracle User Group Tech’17 Conferenc](http://tech17.ukoug.org/)e where I’ll take part in a few Oracle ACE Director briefing and social events, catch-up with some old friends and past colleagues and [speak on the Tuesday](http://tech17.ukoug.org/default.asp?p=16630&dlgact=shwcti&prs_prsid=18556&day_dayid=118) about something that’s fascinated, newly-motivated and quite honestly surprised me in this year away from Oracle BI consulting: the world of [high-growth eCommerce and tech SaaS startups](https://www.mckinsey.com/business-functions/marketing-and-sales/our-insights/the-sales-secrets-of-high-growth-companies) and how they intensively-use analytics, stats models and now AI to out-think, out-manouvre and out-compete their old-world brick-and-mortar competition … and how most of us have never even heard of the tools and analysis techniques they use to do it. * * * In my previous life the BI tool of choice was [Oracle BI Enterprise Edition](https://www.amazon.co.uk/Oracle-Business-Intelligence-Developers-Guide/dp/0071798749), or maybe Tableau, Qlik or Oracle DV if you wanted something self-service or more vizualization-focused. In the eCommerce world it’s [Google Analytics](https://analytics.google.com/analytics/web/provision/?authuser=0#provision/SignUp/), not the free version you’ve used in the past to see how popular your blog is but instead the full Google Analytics 360 Suite which well goes beyond simple counts of page hits and returning visitors. Google Analytics 360 provides digital markers with tools to measure retention and churn, goals achieved and attribution of revenue based on first click, last click and behavioural rules, segment their customers into simple cohorts of customers similar to themselves and advertise to them using Google’s Adwords service in order to land them as prospects, move them rapidly through the purchase funnel to buy something, and with a bit of luck come back again later and buy something else again and again. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2F60d67-34fb8a2d-073e-494b-b5bf-d73ae3d1dbf8_800x468.png%3Fssl%3D1) Google Analytics 360 is the de-facto standard for web analytics but other specialist tools and vendors exist alongside or as specialist competition such as MixPanel and Amplitude Analytics for product analysts, and tech startups such as [Qubit](http://www.qubit.com/) with the Live Tap product that I’m product manager for that marketers use along with the Looker BI tool to make their use of Qubit’s personalization platform more data-driven and productive. * * * What to me makes all of this really interesting is the degree to which the business models and decisions taken by these eCommerce and SaaS businesses are driven by data and analytics, to the extent where [marketing in the eCommerce world is math](https://www.linkedin.com/pulse/20141015104035-5704515-does-marketing-require-math-and-other-stupid-questions/) and [anyone moving into digital marketing analytics needs to know statistics or the market will kill them all](https://zedshaw.com/archive/programmers-need-to-learn-statistics-or-i-will-kill-them-all/), except this time it’s true rather than being something Zed Shaw in his famous blog posts wishes were true. In the presentation next week I’ll talk about how [A/B testing](https://en.wikipedia.org/wiki/A/B_testing), [multivariate testing](https://en.wikipedia.org/wiki/Multivariate_testing) and more recently [contextual multi-armed bandit](https://en.wikipedia.org/wiki/Multi-armed_bandit) models are used to optimize eCommerce sales operarations and mobile app experiences to maximize conversions and select what features to offer, and how bayesian statistical models are commonly employed against raw transaction data to predict with confidence the actual uplift and revenue from a campaign by considering the probability of those transactions based on prior knowkedge and thereby filtering out the one or two outlier big-spenders who only appear once-in-a-while … except if you’re in the eGaming vertical where [those are exactly the customers you’re all fighting over](https://www.recode.net/2014/2/26/11623998/a-long-tail-of-whales-half-of-mobile-games-money-comes-from-0-15) and you certainly don’t want to ignore their activity. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2Fc7815-77ed324b-f1bb-4d3b-b788-9073c5420466_794x541.png%3Fssl%3D1) And, as Qubit’s found out a while ago and the most forward-looking brands who work with them are also discovering, just randomly flipping the colour of a button or blindly discounting doesn’t increase customer lifetime value over the long-term and for that you need to move to truly 1:1, personalized marketing and communications driven by individual-level behavorial event data powered by next-generation big data cloud platforms that are engine[ered from the outset for this purpose, not](https://cloudplatform.googleblog.com/2017/11/how-Qubit-and-GCP-helped-Ubisoft-create-personalized-customer-experiences.html) hacked-together from a bunch of old acquisitions. More imporantly though the next problem has come along and it’s that everyone’s now using their smartphones to access those retailers’ mobile websites but then transacting far less than they did when using desktop browsers because [mobile doesn’t work as well for browsing those retailers’ product catalogs](http://www.qubit.com/sites/default/files/press-releases/Qubit%20Aura%20Announcement%20-%20OCT17.pdf); have you ever found something you like, purely by chance, when trying to buy something new to wear on a mobile phone? Solving problems like this are what’s driving the [deployment of machine learning algorithms against vast amounts of consumer data for product discovery and recommendations](https://martechtoday.com/qubit-launches-instagram-like-ai-powered-product-news-feed-mobile-205333) as led by Qubit and their recently launched Aura product described in in the Youtube video below Qubit Aura recommends new and relevant items to visitors based on their preferences and behavior and then presents it all using an Instagram/Tinder-style interface designed specifically for mobile devices, and in-particular the way we “window-shop” on our phone before transacting either on that same phone at a later date, or back at our desk once we’re in the office or at one of that brand’s stores on the way home from work or at the mall at the weekend. * * * Interestingly though and bringing it back to Oracle and the audience for the presentation I’m giving on Tuesday, Oracle are also now adding AI and data-driven features into their HR, CX, CRM and other line-of-business SaaS apps in an initiative called [Oracle Adaptive Intelligent Applications](https://www.oracle.com/artificial-intelligence/embedded-ai.html) that’s headed up by Jack Berkowitz, who many of you will know from the Oracle BI Applications days and who recently [came on the Drill to Detail Podcast](https://www.drilltodetail.com/podcast/2017/6/13/drill-to-detail-ep30-ai-ml-oracles-adaptive-intelligence-applications-with-special-guest-jack-berkowitz) to talk about Oracle’s investments in this area; and also explained by Oracle’s Mark Hurd at the product announcement session at this year’s Oracle Openworld shown in the video below. Oracle are solving a different problem to Qubit and like [Salesforce with their Einstein initiative](https://www.salesforce.com/blog/2016/09/introducing-salesforce-einstein.html) the aim here is to enable AI for the traditional, non-tech and established business sector who need all of this to work with their payroll, database and manufacturing systems and want Oracle to leverage all the financial and operational data it holds for them to increase their efficiency and make more effective decisions. So … believe it or not that’s just a small part of [what I’ll be talking about at UKOUG Tech’17 next Tuesday in Birmingham](http://tech17.ukoug.org/default.asp?p=16630&dlgact=shwcti&prs_prsid=18556&day_dayid=118). Slides for the session are below: [https://www.slideshare.net/markrittman/analytics-is-taking-over-the-world-again-ukoug-tech17](https://www.slideshare.net/markrittman/analytics-is-taking-over-the-world-again-ukoug-tech17) and in the meantime, if you see me around in one of the other conference sessions or at the Tap & Spile with the other Oracle ACEs and ACE Directors, say hello and I’m always up for a chat about this stuff and how it applies to Oracle customers and partners. --- # Query Federation Comes to Looker 5 with New Data Merge Feature Canonical: https://rittmananalytics.com/blog/2017/11/24/query-federation-comes-to-looker-5-with-new-data-merge-feature URL: https://rittmananalytics.com/blog/2017/11/24/query-federation-comes-to-looker-5-with-new-data-merge-feature Description: Hadoop discovers OLAP – God this makes me feel old, next they’ll discover business metadata layers and indexing [http://t.co/4bwaVEEjO0](http://t.co/4bwaVEEjO0) > > — Mark Rittman (@markrittman) [August 31, 2015](https://twitter.com/markrittman/status/638311769556025344?ref_src=twsrc%5Etfw) … yes it does make me feel old, but it’s great to see such a powerful concept as multidimensional OLAP storage and dimensional models being rediscovered by the big data and startup worlds. --- # Using Google BigQuery, Google Cloud Natural Language API and Looker to Work-Out Exactly How Much… Canonical: https://rittmananalytics.com/blog/2017/07/31/2017-07-31-2017-07-31-using-google-bigquery-google-cloud-natural-language-api-and-looker-to-work-out-exactly-how-much URL: https://rittmananalytics.com/blog/2017/07/31/2017-07-31-2017-07-31-using-google-bigquery-google-cloud-natural-language-api-and-looker-to-work-out-exactly-how-much Description: Back when the incident with my WiFi kettle went viral on Twitter and ended up in the national newspapers, on Radio 4’s The Today Programme and as one of the… Author: Mark Rittman Published: 2017-07-31T00:35:02+00:00 Modified: 2017-07-31T00:35:02+00:00 --- ![Using Google BigQuery, Google Cloud Natural Language API and Looker to Work-Out Exactly How Much…](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2Fcfccd-e13178a8-90bd-4365-920a-82d3482247b0_800x610.png%3Fssl%3D1) Back when the [incident with my WiFi kettle](https://medium.com/mark-rittman/the-story-behind-the-ikettle-the-eleven-hour-struggle-to-make-a-cup-of-tea-and-why-it-was-all-769144d12d7) went viral on Twitter and ended up in the national newspapers, on Radio 4’s The Today Programme and as one of the questions on Have I Got News For You, I scraped the reader comments from the Guardian article that first broke the story along with the Mail Online follow-up article that put a more negative spin on it (“Couldn’t he just flip the switch”, etc). ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2Fcfccd-e13178a8-90bd-4365-920a-82d3482247b0_800x610.png%3Fssl%3D1) I thought it’d be interesting to prep and then load all the comments in Google BigQuery using Google’s new Cloud Dataprep tool, then analyze it all using Google’s Natural Language API and some geocoding libraries to find out exactly how much, and at what intensity, Daily Mail readers hated me and what parts of the country I ought to steer clear of for a while, or at least be prepared to make a cup of tea using a kettle that I have to walk over to and switch on. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2F6d296-da5641ae-a5bf-431a-8985-e2182efda21c_800x557.png%3Fssl%3D1) Visualizing the data that came out of the exercise using Looker, the BI tool I use day-to-day in my role as Product Manager for an analytics service running on Google Cloud Platform, the first thing I spotted was the significant difference in sentiment between Guardian readers (green in the chart below) and Mail Online (red), where you can see the Guardian picking up the story first and comments being typically neutral or positive, whereas the Mail’s readers hated the story almost right from the start and hated it even more the next morning. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2Fe3cb4-e2185ea9-9f62-46cf-ac48-f00b3b01dca9_800x693.png%3Fssl%3D1) As the Mail Online lists the reader’s location alongside each comment I was able to generate latitude and longitude map references for most of them, and mapping them onto a Looker map visualization shows exactly which parts of the UK hated me the most that day. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2Fa6f3c-e2868d8b-8038-4f37-a79b-f206e138a7ea_800x657.png%3Fssl%3D1) Clearly The Mail Online’s Scottish readers typically weren’t impressed with this English innovation, and nor were the Mail’s ex-pat readers down-under who seemingly felt the same way. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2Fe26af-f6253106-b9b8-422a-87e9-f2a8d05063d3_800x668.png%3Fssl%3D1) Once the Mail Online readers’ woke-up to the story in the United States they had a whole new level of disgust with that geek and his rubbish WiFi kettle. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2Fc12a6-1a3de599-51b2-41c0-b273-8825edea7d10_800x607.png%3Fssl%3D1) I was about to blame Piers Morgan for this obvious change of attitude towards us Brits but then spotted, reassuringly, that they seemed to hate the Mail Online even more — all is apparently not lost with our American cousins. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2F748eb-0fd16f8c-6ebb-4c60-882c-6201306e32a5_800x715.png%3Fssl%3D1) So back to the UK. At postcode area level, then, which parts of the UK are a no-go area for a “[Geek \[who\] buys himself a WiFi kettle then spends 11 HOURS trying to make it boil … to work alongside other devices in his £550,000 home.](http://www.dailymail.co.uk/news/article-3834056/Couldn-t-just-flip-switch-Geek-buys-WiFi-kettle-spends-11-HOURS-trying-make-boil.html#ixzz4oMENYGtK)” — nice touch from the Mail journalist there, my wife was actually more annoyed they quoted the price we paid for it several years ago whereas I was hoping they’d bring in that other Daily Mail obsession about [whether something does, or does not give you cancer](https://www.google.co.uk/search?q=daily+mail+house+prices+does+it+cause+cancer&oq=daily+mail+house+prices+does+it+cause+cancer&aqs=chrome..69i57.6332j0j4&sourceid=chrome&ie=UTF-8) through in this case the kettle’s WiFi base-station. Anyway, Hampshire’s a no-go area for me now, along with parts of Wales and the West Midlands although I’m safe in North Norfolk and Humberside at least for now. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2Fed8de-6a3e7ab2-e7bf-45a4-a716-8dcb7dab6218_800x680.png%3Fssl%3D1) If it all gets too much for me in the UK I can at least take refuge in Hong Kong, a place where they obviously appreciate innovative kitchen appliances integrated with other appliances through python scripts that work until the first device gets a new IP address when you unplug and then plug it in again to move it across the room. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2F26d92-c4f3e1c6-1f2a-4e13-9df6-5fc6468c67ac_757x729.png%3Fssl%3D1) Though I need to get there via some means other than stowing away on a container ship bound for Shanghai from the ports at Gateshead and Southampton, though they’re still a better option than hitching a ride on a trawler from Blackpool, the official centre-of-hate for this story in the UK. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2F200dc-0020200b-f6c3-4a28-bb7f-e2dacf32a30b_687x653.png%3Fssl%3D1) Mail Online readers from Blackpool’s dislike for the story was surpassed only by the intensity of that sentiment, another metric that I could get out of the [Google Cloud Platform NLP API](https://cloud.google.com/natural-language/docs/basics) that helps you understand the strength of the feeling someone had when expressing an opinion, not just the words they used. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2F066a6-93cd551a-9231-41d0-9527-3fa7c9b3947e_800x541.png%3Fssl%3D1) So how did this all look when I swapped out Mail Online readers for the five thousand or so Twitter users that retweeted, commented on or otherwise helped spread the story around the Internet those two days back in October 2016? The story started off from the UK, obviously, but with a negative spin that over time, as it travelled around the world became more positive… ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2Fcc863-e70157ba-f150-42a0-b7d5-f89334145808_800x430.png%3Fssl%3D1) … although the Australians clearly still weren’t impressed. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2F89657-b76ea2db-c31c-4927-a460-51148a87e140_800x713.png%3Fssl%3D1) If you’re in London this Tuesday evening and interested to know more about the work I do with BigQuery and Looker in my Product Management role at [Qubit](http://www.qubit.com/), come along to the [Conversion Rate, Analytics and Product Unconference Meetup](https://www.meetup.com/London-Conversion-Rate-Optimisation-Analytics-Product/events/239899482/) where there’s still a few spaces free at the time of writing and I’m one of the three speakers that evening. --- # Google BigQuery, Large Table Joins and How Nested, Repeated Values and the Capacitor Storage Format… Canonical: https://rittmananalytics.com/blog/2017/06/10/2017-06-10-2017-06-10-google-bigquery-large-table-joins-and-how-nested-repeated-values-and-the-capacitor-storage-format URL: https://rittmananalytics.com/blog/2017/06/10/2017-06-10-2017-06-10-google-bigquery-large-table-joins-and-how-nested-repeated-values-and-the-capacitor-storage-format Description: I’m flying over to Hungary early next week to meet my good friend Bence Arato, and to speak at the Budapest Data Forum about analytics and big data on Google’s… Author: Mark Rittman Published: 2017-06-10T15:08:54+00:00 Modified: 2017-06-10T15:08:54+00:00 --- ![Google BigQuery, Large Table Joins and How Nested, Repeated Values and the Capacitor Storage Format…](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2Fe290c-fc830f90-bd9a-4b97-a1d4-4c1f7be73c71_800x546.png%3Fssl%3D1) I’m flying over to Hungary early next week to meet my good friend Bence Arato, and to speak at the Budapest Data Forum about analytics and big data on Google’s Cloud Platform using BI tools such as Looker, both of which I’ve written about on this blog over the past few months. BigQuery lets you store gigabytes and petabytes of data in a distributed, cloud-based query and compute layer that automatically scales-up as you load more data into it, and presents data out in familiar tables and columns, datasets (schemas) and projects (databases) that would be immediately recognisable to anyone [coming into this new technology from the old world of relational databases and SQL queries](https://cloud.google.com/solutions/bigquery-data-warehouse). But that familiarity can be deceiving sometimes, particularly when you start to really do things at scale. Take, for example, two of the sample datasets that Google make available on the [Google BigQuery Public Datasets](https://cloud.google.com/bigquery/public-data/) page; [“Bay Area Bike Share Trips”](https://cloud.google.com/bigquery/public-data/bay-bike-share) that contains records of all public bike hire trips in San Francisco over a number of years: ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2Fe290c-fc830f90-bd9a-4b97-a1d4-4c1f7be73c71_800x546.png%3Fssl%3D1) and another that contains [records of all the San Francisco 311 (non-emergency) calls made since 2008](https://cloud.google.com/bigquery/public-data/sfo-311) including reason for the call, time and caller location, both of which are ready for analysis and can be queried immediately using BigQuery’s ANSI SQL-standard query language. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2F4fb64-b06b37a1-144b-45d9-9f0f-1228e43e2cee_800x538.png%3Fssl%3D1) The bike rides dataset contains just under a million records stored in a 120MB table and returns answers to queries in just a few seconds with none of the indexes, summary tables or in-memory caches that traditional databases and OLAP servers required you to populate and manage to keep query response times within an acceptable limit. In fact, combining these two datasets together so I can see which bikeshare locations had the most traffic accidents associated with them would be an interesting exercise, and BigQuery supports joins between tables using the ANSI join syntax, like this: ``` SELECT zip_code, count(trip_id) as total_trips, call_type, count(call_type) as total_calls FROM `bigquery-public-data.san_francisco.bikeshare_trips` b LEFT JOIN `bigquery-public-data.san_francisco.sffd_service_calls` s ON b.zip_code = s.zipcode_of_incident WHERE call_type = 'Traffic Collision' GROUP BY 1,3 ``` Joining large fact table-style tables together with smaller dimension-style tables works fine with BigQuery, but joining these two large denormalized fact tables together gives us a resources exceeded error after the query runs for over fifteen minutes. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2Fe2af1-0f4274bb-aa64-4aa0-a0d1-c0e0f91607f5_522x117.png%3Fssl%3D1) What we’re seeing here is the trade-off we get when running data warehouse workloads on a distributed query platform like BigQuery. The size of datasets we can work with become huge, but joins between those datasets become relatively expensive due to the need to sort, co-ordinate and send intermediate results between thousands of server nodes in the the cluster. The usual answer is to denormalize even further, creating one single large table containing all the columns from both input tables and multiple rows for each base fact if you’ve got a one-to-many (or many-to-many) relationship between the joined tables. Storage is cheap with cloud platforms from Google and Amazon but BigQuery offers another solution that relies on the [Colossus](http://web.archive.org/web/20160324185413/http://static.googleusercontent.com/media/research.google.com/en/us/university/relations/facultysummit2010/storage_architecture_and_challenges.pdf) distributed file system that Google replaced Google File System with(the inspiration for HDFS), and in-particular the [Capacitor columnar storage format](https://cloud.google.com/blog/big-data/2016/04/inside-capacitor-bigquerys-next-generation-columnar-storage-format) that BigQuery leverages for its fast, column-store table storage * * * Google BigQuery supports several input formats for data you load into tables — CSV files, JSON files, AVRO files and datastore backups — but under the covers BigQuery uses a columnar storage format developed by Google called [Capacitor](https://cloud.google.com/blog/big-data/2016/04/inside-capacitor-bigquerys-next-generation-columnar-storage-format) (originally called ColumnIO) that’s used by Google’s replacement for GFS/HDFS, the Colossus distributed filesystem. Like Parquet and other columnar storage formats that were inspired by ColumnIO, it stores data in compression-friendly columns of nested, repeating values that suit BI-style applications that filter and aggregate big, wide tables of data rather than process transactions using whole rows of data. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2F5149d-9727c5d3-5491-4dd4-9dc5-76d616bbd5e3_690x243.png%3Fssl%3D1) What this means for BigQuery and our two tables of data is that we’d be better off storing the joined table’s values as nested repeating field types in the first table, avoiding the join altogether and storing our data in the way best suited to BigQuery’s storage format. To take an example, if we took the aggregated bike trips values and wrote them out to a JSON format input file like this: ``` {"zipcode":"94537", "trips":[{"start_station_name":"Embarcadero at Vallejo", "avg_duration_sec":8787, "trip_count":"2"}, {"start_station_name":"Townsend at 7th" ,"avg_duration_sec":807, "trip_count":"1"}] ``` that input file could then be used to populate a table that stored each of the trips for a given zipcode as nested repeated values in a single column rather than one row per trip, and with BigQuery’s columnar storage those trips would only be retrieved from storage if that particular column was included in the query, like this: ``` SELECT zipcode, trips.trip_id, trips.duration_sec FROM personal_metrics.bike_trips_nested, UNNEST (trips) as trips LIMIT 10 ``` If I’ve not got input files handy in this nested repeated JSON format, I could use BigQuery Standard SQL to output nested fields using the ARRAY\_AGG(STRUCT() functions, like this: ``` SELECT zip_code as zipcode, ARRAY_AGG(STRUCT(trip_id, duration_sec, start_date, start_station_name, start_station_id, end_date, end_station_id, end_station_name, bike_number)) FROM `aerial-vehicle-148023.personal_metrics.bikeshare_trips` GROUP BY 1 ``` To store the data in this nested format make sure you store the query results in a table, check the Allow Large Results checkbox and deselect the Flatten Results checkbox when using the Web UI. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2Fc01ab-ac18d102-371c-49b0-bd49-a4ac57085129_800x347.png%3Fssl%3D1) You can either then export that tables’ contents into a JSON format output file with nested repeated JSON elements, or query the table in place for more efficient lookups from the repeated dimension values. Taking this to the logical conclusion you could do this for both of the tables we’re looking to join together as they both have these nested records — bike trips per zipcode for one, police incidents per zipcode in the other — and create one big SELECT statement that nests both of these and outputs one big denormalized table with nested repeated column values for each zipcode’s bike trips and incidents records, like this: ``` WITH bike_trips AS ( SELECT zip_code AS zipcode, start_station_name, AVG(duration_sec) AS avg_duration_sec, COUNT(trip_id) AS trip_count FROM personal_metrics.bikeshare_trips b GROUP BY 1,2), incidents_temp AS ( SELECT zipcode_of_incident AS zipcode, call_type, COUNT(call_type) AS call_type_count FROM personal_metrics.sffd_service_calls GROUP BY 1,2) SELECT bt.zipcode, ARRAY_AGG(STRUCT(bt.start_station_name, bt.avg_duration_sec, bt.trip_count)) trips, ARRAY_AGG(STRUCT(ic.call_type, ic.call_type_count)) incidents FROM bike_trips bt LEFT JOIN incidents_temp ic ON bt.zipcode = ic.zipcode GROUP BY 1 ``` and then the equivalent to the two-table join query right at the start would look like this: ``` SELECT zipcode, count(zipcode_trips.trip_count) as total_trips, zipcode_incidents.call_type, count(zipcode_incidents.call_type) as total_calls FROM `aerial-vehicle-148023.personal_metrics.sf_biketrips_incidents_nested` LEFT JOIN UNNEST (trips) as zipcode_trips LEFT JOIN UNNEST (incidents) AS zipcode_incidents WHERE zipcode_incidents.call_type = 'Traffic Collision' GROUP BY 1,3 ``` and then, most importantly, this new query where we’ve eliminated the large table-to-large table join and nested the dimension values in a format that aligns with BigQuery’s storage format, returns results in … 7 seconds. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2Fcd87d-9f77fb46-2fc5-4d9f-b593-4fd8824a7071_800x273.png%3Fssl%3D1) * * * Not all ODBC drivers and BI tools support BigQuery’s nested repeated column format but Looker, the BI tool I’m working with day-to-day on BigQuery right now does, supporting nested joins in LookML like this: ``` explore: sf_biketrips_incidents_nested { join: sf_biketrips_incidents_nested__incidents { view_label: "Incidents" sql: LEFT JOIN UNNEST(${sf_biketrips_incidents_nested.incidents}) as sf_biketrips_incidents_nested__incidents ;; relationship: many_to_one } join: sf_biketrips_incidents_nested__trips { view_label: "Trips" sql: LEFT JOIN UNNEST(${sf_biketrips_incidents_nested.trips}) as sf_biketrips_incidents_nested__trips ;; relationship: many_to_one } } ``` ``` view: sf_biketrips_incidents_nested { sql_table_name: personal_metrics.sf_biketrips_incidents_nested ;; ``` ``` dimension: incidents { hidden: yes sql: ${TABLE}.incidents ;; } ``` ``` dimension: trips { hidden: yes sql: ${TABLE}.trips ;; } ``` ``` dimension: zipcode { type: zipcode sql: ${TABLE}.zipcode ;; } ``` ``` measure: count { type: count drill_fields: [] } } ``` ``` view: sf_biketrips_incidents_nested__trips { dimension: id { primary_key: yes sql: CONCAT(sf_biketrips_incidents_nested.zipcode,${TABLE}.start_station_name) ;; } measure: avg_duration_sec { type: average sql: ${TABLE}.avg_duration_sec ;; } ``` ``` dimension: start_station_name { type: string sql: ${TABLE}.start_station_name ;; } ``` ``` measure: trip_count { type: sum sql: ${TABLE}.trip_count ;; } } ``` ``` view: sf_biketrips_incidents_nested__incidents { dimension: id { primary_key: yes sql: CONCAT(sf_biketrips_incidents_nested.zipcode, ${TABLE}.call_type) ;; } dimension: call_type { type: string sql: ${TABLE}.call_type ;; } ``` ``` measure: call_type_count { type: sum sql: ${TABLE}.call_type_count ;; } } ``` and displaying the two nested columns as regular Looker views in the dashboard, so users don’t even need to be aware that you’ve denormalized and optimized the storage of those two tables and details of bike trips and police incidents are now stored in nested repeated columns. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2F05a34-08af85b2-47c4-424a-8eba-b7b12ddfab63_800x417.png%3Fssl%3D1) Like anything in database design or IT in-general nested columns aren’t a silver bullet that solves every large table join or query optimization problem, and they will hit the same sort of resource limit issues if you try to store thousands or millions of dimension values in this nested repeating format. But for most star-schema style queries across sets of denormalized fact and dimension tables they work better when you store data in this format, and using the ARRAY\_AGG and STRUCT functions you can even generate test data in this more optimized format without having to write your own JSON data exporter to nest the dimension values properly. --- # Analytic Views, Oracle Database 12.2 Canonical: https://rittmananalytics.com/blog/2017/05/29/2017-05-29-2017-05-29-analytic-views-oracle-database-12-2 URL: https://rittmananalytics.com/blog/2017/05/29/2017-05-29-2017-05-29-analytic-views-oracle-database-12-2 Description: An interesting new feature came out with the 12c Release of Oracle Database Enterprise Edition called Analytic Views that looks, at first sight, to be an extension of the existing… Author: Mark Rittman Published: 2017-05-29T22:24:35+00:00 Modified: 2017-05-29T22:24:35+00:00 --- ![Analytic Views, Oracle Database 12.2](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2F81fda-e4dd26c5-c3c0-4da5-9055-90959446d76c_625x383.png%3Fssl%3D1) An interesting new feature came out with the 12c Release of Oracle Database Enterprise Edition called [Analytic Views](https://docs.oracle.com/database/122/DWHSG/overview-analytic-views.htm#RESOURCEID-183323-7FDAC1B6) that looks, at first sight, to be an extension of the existing dimensions and materialized views features the Oracle database has had since the 8i release. For anyone who remembers those features being added to the Oracle Database back in the mid 90’s (and if you don’t, here’s a youthful-looking blogger [writing about that topic](http://www.dba-oracle.com/t_olap_dimensions_cubes.htm) on a soon-to-be-rather-controversial DBA website), the CREATE DIMENSION statement allowed you to define an aggregation rule for use by the database’s query optimizer, like this: ``` CREATE DIMENSION products LEVEL product IS (products.prod_id, products.prod_desc, products.prod_list_price, products.prod_min_price, products.prod_name, products.prod_pack_size, products.prod_status, products.supplier_id, products.prod_unit_of_measure, products.prod_weight_class) LEVEL subcategory IS (products.prod_subcategory, products.prod_subcat_desc) LEVEL category IS (products.prod_category, products.prod_cat_desc) HIERARCHY prod_hier (product CHILD OF subcategory CHILD OF category) ATTRIBUTE product DETERMINES (products.prod_category, products.prod_id) ATTRIBUTE subcategory DETERMINES (products.prod_category, products.prod_subcategory) ATTRIBUTE category DETERMINES products.prod_category; ``` These aggregation rules could be thereafter be used by the optimizer to work out whether to rewrite the users’ query to use a pre-computed aggregate to answer that query instead of rolling-up from the detail-level the user specified in their query. Since then, subsequent releases of the Oracle Database have introduced a separate but related feature, the OLAP Option, that used technology originated from the old Express Server to create what were called Analytic Workspaces, fully-featured OLAP multi-dimensional caches together with their own programming language and suite of applications for sales and budgeting/planning. Over the years Oracle introduced features to integrate analytic workspaces with the SQL query language used by the core database rather than developers having to learn and use OLAP DML, Oracle OLAP’s proprietary programming language, and the dimensional model provided by the OLAP Option and all OLAP servers is just as relevant today as it was back in the 1970’s when that technology first became available, and at that time and until recently the other innovation provided by dedicated multidimensional OLAP Servers — ultra-fast queries made possible by the use of a separate pre-computed and highly-indexed mid-tier data cache — was so valuable to users that the overhead in maintaining these caches, and the query latency they introduced because of the time it took to refresh and recompute the aggregates — made sense. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2F81fda-e4dd26c5-c3c0-4da5-9055-90959446d76c_625x383.png%3Fssl%3D1) Until recently that is, when hardware used to host database servers suddenly became able to host the entire database in-memory and disk I/O was no longer a problem, and fast CPUs with multiple cores and large amounts of on-board cache RAM meant pre-computing calculated measures was no longer needed. So along came Oracle Database 12c and the In-Memory Option, that until the [incident with the WiFi kettle](https://medium.com/@markrittman/graph-analysis-in-the-how-a-tweet-went-viral-conference-presentation-95f62b7dd4cd) was my claim to fame as the person Larry Ellison quoted on stage at Oracle when I said it was so easy to use it was “almost boring”. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2F869c6-91d56c19-6b68-4629-bc44-d8a4d9562540_600x452.png%3Fssl%3D1) Increasingly, it no longer made any sense to define and then maintain a separate multi-dimensional OLAP cache in a sort-of-embedded-server like Oracle OLAP, when the so-easy-to-use-it’s-boring Database In-Memory Option along with materialized views and bitmap indexes were just as fast to return query results to users … especially when this in-memory option also came with another feature, vector transformation plans, that enabled a [special type of in-memory aggregation using the vector processing features in modern CPUs](https://www.rittmanmead.com/blog/2014/08/taking-a-look-at-the-oracle-database-12c-in-memory-option/#comment-2714741393) and had been developed — rather interestingly — by none other than the OLAP Option development team headed up by none other than … Bud Endress, the long-term development lead for Oracle OLAP and before that, Express Server. So what’s going on here then? Well back then this adding of special aggregation capabilities into the in-memory database feature by the old OLAP Server team clearly signalled that this was where the Oracle intended to take the technology legacy from their standalone and then partly-integrated multidimensional OLAP server cache. but then the second shoe dropped back at Oracle Openworld two years ago when Analytic Views were first launched in a presentation by … Bud Endress. > Analytic Views feature coming in 12.2 seems to be SQL-over-Oracle-OLAP, without materialising the OLAP cube [#oow15](https://twitter.com/hashtag/oow15?src=hash&ref_src=twsrc%5Etfw) [pic.twitter.com/1P8UzsfgOj](https://t.co/1P8UzsfgOj) > > — Mark Rittman (@markrittman) [October 29, 2015](https://twitter.com/markrittman/status/659776557523869696?ref_src=twsrc%5Etfw) Analytic Views then came along with the 12.2 release of Oracle OLAP and fairly obviously were the other half to the puzzle of where Oracle were planning to take the people, ideas and inspiration that came out of Oracle OLAP and the twin core user benefits of a dimensional model, and fast response to user queries involving aggregation and slice-and-dice of structured datasets. Not only that though, they also included a new take on the simplified SQL query format that Oracle OLAP introduced a few years ago, gave users the ability to build out a complete Oracle BI-style business model right over the data in the database … and, as if that wasn’t enough surprises, support (via third-party Simba ODBC drivers) for MDX. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2F1e189-a1e48950-5ffb-444f-b6f9-3e7ee2015a41_800x654.png%3Fssl%3D1) * * * Taking the IoT, wearables and social media data I’ve been [collecting and storing up until now in Google BigQuery](https://medium.com/mark-rittman/what-bi-development-looks-like-with-bigquery-google-cloud-apis-looker-and-fluent-courtesy-of-the-f9e8c23ba0c6) and transferring that across to an Oracle Database 12c 12.2 database with Analytic Views and the In-Memory Option enabled, and using Oracle SQL Developer 4.2 with its [new Analytic Views functionality](https://twitter.com/markrittman/status/864235490265976835) now included, you can see that there is a new dedicated section for Analytic Views in the main interface not unlike the old Analytic Workspace Manager standalone tool from the Oracle 10g/11g days. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2Fd3e5d-323c4174-509c-4145-b44a-366730ecab6e_320x487.png%3Fssl%3D1) Creating a new analytic view involves one of two routes in SQL Developer 4.2 (or you could of course script it all using DDL statements); you can create a quick-and-dirty analytic view over a single table using the Quick Analytic View menu option as per the screenshot below that then introspects your data to determine candidate measure and dimension columns, but this doesn’t seem to handle the sorts of wide, multiple level-per-attribute dimension denormalized tables that are common in distributed query engines like Google BigQuery that prefer those over joining lots of separate tables; ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2F21fac-a85e0056-49e0-46f2-aab7-7c655c753394_505x410.png%3Fssl%3D1) or, as I did for my testing you can normalize your fact and dimension tables so they’re separate and joined via foreign key constraints and then create in turn your attribute dimensions, hierarchies and attribute views containing measures and aggregation rules as shown in the screenshots below. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2F2b386-f109265f-b48a-41e1-9425-edf73e6b159d_758x495.png%3Fssl%3D1) So far, very much in the same vein as creating relational and OLAP Option cube structures in previous database versions and developer tools in the past, and you can view the commands that SQL Developer will then send to the database using one of the dialog tabs, giving you a set of DDL commands looking like this: ``` CREATE ATTRIBUTE DIMENSION DEVICE_DIMENSION DIMENSION TYPE STANDARD CAPTION 'DEVICE DIMENSION' DESCRIPTION 'DEVICE DIMENSION' USING DEVICE_DIM ATTRIBUTES ( DEVICE AS DEVICE CAPTION 'DEVICE' DESCRIPTION 'DEVICE' , DEVICETYPE AS DEVICETYPE CAPTION 'DEVICETYPE' DESCRIPTION 'DEVICETYPE' , ROOM AS ROOM CAPTION 'ROOM' DESCRIPTION 'ROOM' ) LEVEL DEVICE_LEVEL LEVEL TYPE STANDARD CAPTION 'DEVICE LEVEL' DESCRIPTION 'DEVICE LEVEL' KEY ( "DEVICE") MEMBER NAME DEVICE MEMBER CAPTION DEVICE MEMBER DESCRIPTION DEVICE ORDER BY MIN DEVICE ASC NULLS FIRST DETERMINES ( "DEVICE") LEVEL ROOM_LEVEL LEVEL TYPE STANDARD CAPTION 'ROOM LEVEL' DESCRIPTION 'ROOM LEVEL' KEY ( "ROOM") MEMBER NAME ROOM MEMBER CAPTION ROOM MEMBER DESCRIPTION ROOM ORDER BY MIN ROOM ASC NULLS FIRST DETERMINES ( "ROOM") LEVEL DEVICETYPE LEVEL TYPE STANDARD CAPTION 'Devicetype' DESCRIPTION 'Devicetype' KEY ( "DEVICETYPE") MEMBER NAME TO_CHAR("DEVICETYPE") ORDER BY MIN DEVICETYPE ASC NULLS LAST DETERMINES ( "DEVICETYPE") ALL MEMBER NAME 'ALL' MEMBER CAPTION 'ALL' MEMBER DESCRIPTION 'ALL'; ``` ``` CREATE ATTRIBUTE DIMENSION DATES_DIMENSION CAPTION 'DATES DIMENSION' DESCRIPTION 'DATES DIMENSION' USING DATES_DIM ATTRIBUTES ( DATE_DATE AS DATE_DATE CAPTION 'DATE_DATE' DESCRIPTION 'DATE_DATE' , DAY_NAME AS DAY_NAME CAPTION 'DAY_NAME' DESCRIPTION 'DAY_NAME' , DAY_NUMBER AS DAY_NUMBER CAPTION 'DAY_NUMBER' DESCRIPTION 'DAY_NUMBER' , YEAR AS YEAR CAPTION 'YEAR' DESCRIPTION 'YEAR' , YEAR_DAY_OF_YEAR AS YEAR_DAY_OF_YEAR CAPTION 'YEAR_DAY_OF_YEAR' DESCRIPTION 'YEAR_DAY_OF_YEAR' , YEAR_MM AS YEAR_MM CAPTION 'YEAR_MM' DESCRIPTION 'YEAR_MM' ) LEVEL DATE_DATE CAPTION 'Date Date' DESCRIPTION 'Date Date' KEY ( "DATE_DATE") MEMBER NAME TO_CHAR("DATE_DATE") MEMBER CAPTION TO_CHAR("DATE_DATE") MEMBER DESCRIPTION TO_CHAR("DATE_DATE") ORDER BY MIN DATE_DATE ASC NULLS FIRST DETERMINES ( "DAY_NAME" , "DAY_NUMBER" , "YEAR_DAY_OF_YEAR") LEVEL YEAR CAPTION 'Year' DESCRIPTION 'Year' KEY ( "YEAR") ALTERNATE KEY YEAR MEMBER NAME YEAR MEMBER CAPTION YEAR MEMBER DESCRIPTION YEAR ORDER BY MIN YEAR ASC NULLS FIRST LEVEL YEAR_MM CAPTION 'Year Mm' DESCRIPTION 'Year Mm' KEY ( "YEAR_MM") MEMBER NAME YEAR_MM MEMBER CAPTION YEAR_MM MEMBER DESCRIPTION YEAR_MM ORDER BY MIN YEAR_MM ASC NULLS FIRST ALL MEMBER NAME 'ALL' MEMBER CAPTION 'ALL' MEMBER DESCRIPTION 'ALL'; ``` ``` CREATE HIERARCHY DATES_HIERARCHY CAPTION 'DATES_HIERARCHY' DESCRIPTION 'DATES_HIERARCHY' USING DATES_DIMENSION ( "DATE_DATE" CHILD OF "YEAR_MM" CHILD OF "YEAR" ); ``` ``` CREATE HIERARCHY DEVICE_TYPE_HIERARCHY CAPTION 'DEVICE_TYPE_HIERARCHY' DESCRIPTION 'DEVICE_TYPE_HIERARCHY' USING DEVICE_DIMENSION ( "DEVICE_LEVEL" CHILD OF "DEVICETYPE" ); ``` ``` CREATE HIERARCHY ROOM_HIERARCHY CAPTION 'ROOM_HIERARCHY' DESCRIPTION 'ROOM_HIERARCHY' USING DEVICE_DIMENSION ( "DEVICE_LEVEL" CHILD OF "ROOM_LEVEL" ); ``` ``` CREATE ANALYTIC VIEW IOT_DEVICES_AV CAPTION 'IOT_DEVICE_READINGS' DESCRIPTION 'IOT_DEVICE_READINGS' USING IOT_DEVICES DIMENSION BY ( DEVICE_DIMENSION AS DEVICE_DIMENSION KEY ( "DEVICE") REFERENCES ( "DEVICE") HIERARCHIES ( DEVICE_TYPE_HIERARCHY AS DEVICE_TYPE_HIERARCHY DEFAULT, ROOM_HIERARCHY AS ROOM_HIERARCHY ), DATES_DIMENSION AS DATES_DIMENSION KEY ( "DATE_TIME") REFERENCES ( "DATE_DATE") HIERARCHIES ( DATES_HIERARCHY AS DATES_HIERARCHY DEFAULT ) ) MEASURES ( VALUE FACT VALUE CAPTION 'VALUE' DESCRIPTION 'VALUE' ) DEFAULT MEASURE VALUE DEFAULT AGGREGATE BY AVG; ``` But the bit that’s to my mind interesting in the Analytic View creation dialog is the Cache Measure Groups option in the left-hand menu of the SQL Developer screen. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2Fbb658-08c62ebf-e4e1-41d8-91b0-317b9deb5756_785x604.png%3Fssl%3D1) This, and the vector transformation feature introduced with Oracle Database 12.1, are the key to how Analytic Workspaces will return aggregated query results as fast, if not faster, than using a dedicated multidimensional OLAP cache layer. Optimisations in the SQL parser for analytic views that on one hand enable simplified SQL queries without the need for GROUP BY or join clauses, like this: ``` SELECT time_hier.member_name as TIME, sales, sales_prior_period FROM sales_av HIERARCHIES(time_hier) WHERE time_hier.level_name IN ('YEAR','QUARTER') ORDER BY time_hier.hier_order; ``` also use these cache measure group settings to hard-wire the Analytic View into any pre-created materialized views containing aggregations of your measures, side-stepping the usual query rewrite mechanism used by the database query optimizer to ensure, along with vector transformations, that users have split-second response times to their queries whether at detail-level or aggregated across one or more of the analytic view hierarchies. You can [try this feature out for yourself](https://livesql.oracle.com/apex/livesql/file/content_D1WFQFUZG1RIXI6CAETTGK2LH.html) using Oracle’s Live SQL demo environment here, and it’s important to understand what this feature and the integration of analytic views with the in-memory database option with its vector transformation feature really means — OLAP aggregation rethough for today’s in-memory database servers. So this, in my opinion, is where Oracle are headed in terms of meeting the needs of database customers who want fast, easy-to-maintain OLAP query functionality that provides the response time of a dedicated OLAP server but does it through leveraging the in-memory capabilities of Oracle Database 12c, the people and ideas from the OLAP Option product and the huge amounts of memory and compute capacity you get in modern servers. All very interesting, and interesting as well to see where the high-end traditional relational database vendors are taking OLAP analysis now old-school OLAP servers have had their day but dimensional analysis and dimensional business models are still just as relevant. --- # What BI Development Looks like with BigQuery, Google Cloud APIs, Looker and Fluentd (courtesy of… Canonical: https://rittmananalytics.com/blog/2017/05/01/2017-05-01-2017-05-01-what-bi-development-looks-like-with-bigquery-google-cloud-apis-looker-and-fluentd-courtesy-of URL: https://rittmananalytics.com/blog/2017/05/01/2017-05-01-2017-05-01-what-bi-development-looks-like-with-bigquery-google-cloud-apis-looker-and-fluentd-courtesy-of Description: For the last six months I’ve been working in London helping build out an analytics service built on Google’s Cloud Platform, BigQuery and the Looker BI tool. It’s been a… Author: Mark Rittman Published: 2017-05-01T00:39:02+00:00 Modified: 2017-05-01T00:39:02+00:00 --- ![What BI Development Looks like with BigQuery, Google Cloud APIs, Looker and Fluentd (courtesy of…](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2F1c650-caf89588-d1dc-412f-8b24-c46e1827e59e_775x517.png%3Fssl%3D1) For the last six months I’ve been working in London helping build out an analytics service built on Google’s Cloud Platform, BigQuery and the Looker BI tool. It’s been a very interesting and rewarding experience working within product management in an engineering team using agile development practices, developing with node.js and clojure and shipping releases every few months. However, working in product management rather than in development or consulting means I’m more likely to be creating product roadmaps and working on partner enablement than developing using BigQuery and Looker, so I try and use the journey time up to London each day to do some development myself, work out what development requests and users stories would be easy or hard to have our engineering team deliver … and because as a techie and career-long BI and data warehousing developer this stuff is just too cool and too interesting to not get developing with. * * * So over the past few months I’ve been taking the [IoT and wearables data analysis system I put together using Hadoop](https://medium.com/mark-rittman/the-story-behind-the-ikettle-the-eleven-hour-struggle-to-make-a-cup-of-tea-and-why-it-was-all-769144d12d7) running on servers back home and [ported it to Google BigQuery](https://medium.com/mark-rittman/google-bigquery-and-why-big-data-is-about-to-have-its-gmail-moment-39f3c6308d55) and more recently [Looker](https://looker.com/) running in the cloud, using [Fluentd](http://www.fluentd.org/) as the log aggregation engine running on a [Google Compute Engine](https://cloud.google.com/compute/)\-hosted VM to collect, process and ship to BigQuery all the data from wearable devices, social media and other services along with IoT event data from SmartThings that I previously fed into logstash and Hadoop. Adding Google Cloud Storage for staging incoming data extracts from services such as [Moves](https://www.moves-app.com/) and [Google Takeout](https://takeout.google.com/settings/takeout) and running it all in Google Cloud Service, this updated version of my analytics architecture looks like the diagram below. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2F1c650-caf89588-d1dc-412f-8b24-c46e1827e59e_775x517.png%3Fssl%3D1) Typical cost for running a VM hosting a simple Fluentd server and running all month is around $80. BigQuery is more or less free for developer use with [charging based largely on how much you query](https://cloud.google.com/bigquery/pricing) with some costs for streaming inserts and data storage, but with the first 1TB of queries free each month and minimal storage needs, Google’s cloud data warehouse platform has pretty-much taken over from Oracle as my day-to-day development platform because, like Oracle did in the days of OTN downloads of on-premise developer tools and databases with permissive licensing for personal training and developing prototype applications, you can learn the technology essentially for free and really get to know and understand the tools without worrying about short trial license periods or having to pay thousands of dollars for full commercial licenses. * * * [I’ve written](https://medium.com/mark-rittman/google-bigquery-and-why-big-data-is-about-to-have-its-gmail-moment-39f3c6308d55) and [presented on Google BigQuery](https://www.slideshare.net/markrittman/new-world-hadoop-architectures-what-problems-they-really-solve-for-oracle-dbas) a few times since being introduced to it late last year so I won’t go into how this distributed, column-store query engine based on [Google’s Dremel engine](https://research.google.com/pubs/pub36632.html) works, but there’s another technology I used as part of this round of development, an open-source technology called Fluentd that I used instead of Flume and Logstash for the log aggregation part of this project that I’ve found particularly useful given its range of input and output data source plugins, and its support for PubSub, BigQuery and other parts of the [Google Cloud Platfor](http://www.fluentd.org/plugins#google-cloud-platform)m making it easy to accept data from IFTTT and SmartThings and then stream it, lightly transformed and consistently timestamped, into a set of BigQuery tables. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2F7c6f3-b31ccce3-ab50-4849-b79c-c0e4afebef5a_800x617.png%3Fssl%3D1) For example, creating a data feed that brought in metrics on car journeys via the [Dash IoS app](https://itunes.apple.com/gb/app/dash-smarter-driving-every-day/id904108101?mt=8) and a small bluetooth device that plugs into your car’s diagnostics port involves first registering an IFTTT applet to trigger when that app registers a journey complete event, with the applet payload then being an [IFTTT Maker webhook](https://ifttt.com/maker_webhooks) that sends the journey metrics to Fluentd in JSON format as an HTTP POST request. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2F5a635-9b28b3dd-5967-4dea-bde3-178985aa4b6b_800x521.png%3Fssl%3D1) Fluentd then receives the JSON document using its [HTTP Input plugin](http://docs.fluentd.org/v0.12/articles/in_http), processes, transforms and timestamps the record and then sends it as a streaming insert into a one of the tables I’ve setup in Google BigQuery, over which I’ve then created a SQL view that transforms all of IFTTT’s odd “3h 5m” and “August 14, 2004 at 10:15PM” date and time formats into additional timestamps that Looker and other BI tools can automatically parse into date and time elements. ``` select date_time, timestamp_trunc(date_time,DAY) as date_day, PARSE_TIMESTAMP('%B %d %Y %I:%M%p',REPLACE(REPLACE(StartedAt,', ',' '),' at','')) as journey_start_date_time, PARSE_TIMESTAMP('%B %d %Y %I:%M%p',REPLACE(REPLACE(EndedAt,', ',' '),' at','')) as journey_end_date_time, TIMESTAMP_DIFF(PARSE_TIMESTAMP('%B %d %Y %I:%M%p',REPLACE(REPLACE(EndedAt,', ',' '),' at','')), PARSE_TIMESTAMP('%B %d %Y %I:%M%p',REPLACE(REPLACE(StartedAt,', ',' '),' at','')), MINUTE) AS journey_mins, StartAddress, StartWeatherConditions, StartLatitude, StartLongitude, DrivingScore, cast(replace(DistanceDrivenWithLabel,' Km','') as FLOAT64) as distance_km, EngineLightAlerts, HardBrakeAlerts, HardAccelerationAlerts, SpeedAlerts, cast(replace(FuelConsumedWithLabel,' l','') as FLOAT64) as fuel_l, cast(replace(AvgDistancePerHourWithLabel,' kph','') as FLOAT64) as speed_kph, cast(replace(AvgFuelConsumptionWithLabel,' kpl','') as FLOAT64) as fuel_econ_kpl, EndLatitude, EndLongitude, EndAddressMapURL from `xxxxx.personal_metrics.fluentd_dash_journeys` ``` One of the nice things you can do with Fluentd is call arbitrary ruby scripts to enrich incoming data via the [Fluentd](https://github.com/SNakano/fluent-plugin-script) script plugin, something I’ve used to analyze for sentiment and key entities all incoming tweets, emails, article saves and other social media and other communications using [Google’s new Natural Language Processing API](https://cloud.google.com/natural-language/). To do this I pass all incoming records of these types through a filter definition in the Fluentd config file, like this… ``` type script path /home/mark/gcp_nlp/nlp_process.rb ``` …with that script then using the [Ruby Cloud Natural Language Client](https://googlecloudplatform.github.io/google-cloud-ruby/#/docs/google-cloud-language/v0.26.2/google/cloud/language) to calculate the sentiment score and magnitude of that sentiment, extract the entities from the incoming text body and then pass these as new Fluentd record keys back to the output plugin to write as additional columns into the BigQuery table I used to store all these communication events. ``` def configure(conf) super require "google/cloud/language" end def start super end def shutdown super end def filter (tag, time, record) require "google/cloud/language" project_id = "xxxxxxxx" language = Google::Cloud::Language.new project: project_id text = record["post_body"] document = language.document text sentiment = document.sentiment entities = document.entities record["sentimentscore"] = "#{sentiment.score}" record["sentimentmagnitude"] = "#{sentiment.magnitude}" entitylist = "" entities.each do |entity| entitylist = entitylist + "#{entity.name} " end record["entityname"] = "#{entitylist.strip}" record end ``` Fluentd can also [collect the device and sensor events](https://github.com/jt55401/smartthings-datalogger/blob/master/smartthings-datalogger.groovy) from my SmartThings hub using a groovy [“smartapp”](https://support.smartthings.com/hc/en-gb/articles/205380504-SmartApps) running in Samsung’s SmartThings backend cloud service. All that’s left to do is join the incoming hub events to a lookup table that classifies the IoT events as sensors, switches, colour settings and other event types and groups them by room and you’ve got a useful set of home enviroment and activity data to join up with the other dataset. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2F536c3-88545d45-fc6f-47cc-877d-94b725591fe7_800x320.png%3Fssl%3D1) Other services such as [Moves](https://www.moves-app.com/) have APIs that I connect to every few hours using clients [such as this one](https://github.com/lysol/moves) also on Github that runs on the same Google Compute Engine VM that runs the FluentD log collector, with new movement data copied across to a Google Cloud Storage bucket and inserted every few hours into another BigQuery table ready for analysis. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2F8ac0c-98e80791-4aab-431a-a1aa-5478b62b4949_800x405.png%3Fssl%3D1) I’ve also been collecting weight readings and sleep readings but these can be sporadic, with weight collected via a Withings WiFi scale but often only every few days, whilst sleep time in minutes should be recorded every day but sometimes I forget to wear my health band, so I used BigQuery SQL’s [analytic windowing functions](https://cloud.google.com/bigquery/docs/reference/standard-sql/functions-and-operators#analytic-functions) to calculate my average weight over 7 days (a good approach anyway as your weight can fluctuate day-to-day), and over three days for sleep. ``` SELECT date_time, AVG(weightKg) OVER(ORDER BY ts RANGE BETWEEN 604800 PRECEDING AND CURRENT ROW) AS weightKg_avg_7_days, AVG(FatMassKg) OVER(ORDER BY ts RANGE BETWEEN 604800 PRECEDING AND CURRENT ROW) AS FatMassKg_avg_7_days FROM ( SELECT TIMESTAMP_TRUNC(d.date_time,DAY) as date_time, avg(w.weightKg) as weightKg, avg(w.FatMassKg) as FatMassKg, UNIX_SECONDS(TIMESTAMP_TRUNC(d.date_time,DAY)) AS ts FROM `aerial-vehicle-148023.personal_metrics.date_dim` d LEFT OUTER JOIN `xxxx.personal_metrics.fluentd_weighings` w ON d.date_time = TIMESTAMP_TRUNC(w.date_time,DAY) GROUP BY d.date_time, ts ) order by date_time asc ``` These two views are then left-outer joined to the main health daily view so that I’ve got weight and sleep readings every day alongside step count, calories burned and other metrics from my Jawbone UP health band. So where this leaves me now is at the point where I’ve got a pretty interesting dataset running in a Google BigQuery cloud-hosted column-store data warehouse, with the data updated in real-time and enriched and transformed ready for analysis. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2F1d896-77cef649-091e-41cc-a8a2-8885201acba5_321x532.png%3Fssl%3D1) For now though the thing that’s made this all really interesting is joining it all up and then analyzing it as one big dataset using Looker, another tool I talked about recently on this blog and one which re-introduces an old concept familiar to BI veterans, the “semantic model”, to this new world of distributed, cloud-based big data analytics. * * * Looker the company and Looker the BI tool are both attracting a lot of attention right now after [Google’s recent investment](https://www.forbes.com/sites/alexkonrad/2017/03/30/looker-bi-raises-81-5m-alphabet/#767d75387bd4), and the general buzz around big data analytics running on Google and Amazon’s cloud platforms. I’ve been working with Looker now for around six months as the preferred BI tool for the platform I’m working with, and Looker’s philosophy towards analytics together with the parallels I can see between the enterprise semantic models I used to create working with tools such as Oracle BI and Looker’s updated take on that idea led me to sign-up as a developer partner with Looker in order to get to understand their platform and technology even closer, the same thought process I went through just over 10 years ago when Oracle acquired Siebel and [introduced me to Siebel Analytics](https://www.rittmanmead.com/blog/2006/01/siebel-analytics-the-jewel-in-the-project-fusion-crown/) and what became the focus of my career and the [subject of two books](https://www.amazon.co.uk/Mark-Rittman/e/B00B8YEQMU), what in-time became Oracle Business Intelligence. To be clear and with the benefit of having worked with BI tools for almost 20 years Looker the BI tool is still very primitive in many ways — in terms of data visualization options and general UI and usability it’s frankly, pretty basic and it’s quirky use of terminology such as “looks” (for reports), “explores” (for subject areas) and “views” (for tables) and many activities we’re used to being simple and intuitive in existing BI tools such as creating and saving catalogs of reports are non-obvious to end-users or just plain missing … but crucially, all of these missing features or UI quirks can be iterated on in a SaaS product and instead, Looker have got two things right: - Their SQL query generation engine was designed from day one to work with distributed query engines like BigQuery - They’ve taken the concept of a semantic model, data model abstraction and combining enterprise datasets but crucially, done it in a way that software engineers and data analysts working in their target market can relate to Looker was designed from the start to query the underlying data source directly rather create a mid-tier cache and query that instead of the underlying data source; Tools like Tableau and Qlikview did well for many years running their user queries of a local cache on the users’ desktop and gave those users much faster response times than they were used to with old, enterprise BI tools that tried to speed up queries by creating layers of aggregate tables, but this approach just isn’t practical with massive, constantly-updated big data sets Looker also generates SQL queries aware of the way platforms like BigQuery charge for their use — by the amount of data queried rather than server processors or named users. BigQuery is a column-store database that brings back data only for those columns you ask for, but tools like Tableau routinely request all columns for a table making them impractical to use for these types of data stores; Looker also aggressively elimates joins from queries and [supports the nested column approach](https://discourse.looker.com/t/nested-data-in-bigquery-repeated-records/4174) preferable for distributed data stores like BigQuery where joins are much more expensive to process than we’re used to with relational databases Finally, Looker has introduced a concept very familiar to traditional BI developers but new to most developers working on big data analytics projects, the concept of a semantic model, metadata layers and data abstraction; importantly though, the approach they’ve taken in LookML is designed to appeal to the actual developers building out these types of systems today — software engineers working on web and big data projects who write code, use modern development lifecycles and version everything in distributed version control system such as Git. So to take the dataset I put together in BigQuery, the first step in the development process with Looker is to create what’s termed “views” for each of the underlying BigQuery table or view sources; in the example below you can see an (abbreviated, there’s more measure definitions in the full view definition file) definition of the “health stats” LookML view, and things to note in this modelling approach are (1) it’s all code, there’s no GUI for developing this all graphically, which suits software engineers just fine as it’s easier from day one to code, version and diff (compare) with other code releases; (2) the dimension\_group we define at the start takes care of extracting all the different time periods out of a BigQuery timestamp datatype, and (3) the measure type definitions of sum, average and so on are there to give Looker more flexibility in aggregating (and symetrically aggregating, what we used to call fan-trap avoidance in the old days) measure values. ``` view: v_health_stats { sql_table_name: rittman_dw.v_health_stats ;; ``` ``` dimension_group: date_day { type: time hidden: yes timeframes: [ raw, time, date, week, month, quarter, year ] sql: ${TABLE}.date_day ;; } ``` ``` dimension: pk { type: string hidden: yes primary_key: yes sql: ${TABLE}.pk ;; } ``` ``` measure: total_active_mins { type: sum sql: ${TABLE}.active_mins ;; value_format_name: decimal_2 } ``` ``` measure: avg_sleep_mins_avg_3_days { type: average label: "Avg Sleep in Mins" sql: ${TABLE}.sleep_mins_avg_3_days ;; value_format_name: decimal_2 } ``` ``` measure: avg_weightKg_avg_7_days { type: average label: "Avg WeightKg" sql: ${TABLE}.weightKg_avg_7_days ;; value_format_name: decimal_2 } ``` ``` measure: total_distance_km { type: sum sql: ${TABLE}.distance_km ;; value_format_name: decimal_2 } ``` ``` } ``` Once you’ve defined all your views, the next step is to define your model made up of what are termed “explores” — think of models as the equivalent of a universe in Business Objects or a repository in Oracle Business Intelligence, and explores as subject areas linking together through sets of views that join on common columns. ``` connection: "rittman_bigquery" ``` ``` # include all the views include: "*.view" ``` ``` # include all the dashboards include: "*.dashboard" ``` ``` label: "Rittman DW" ``` ``` explore: fluentd_log { label: "Rittman BigQuery DW" view_label: "1 - Home Metrics" join: v_communications { type: left_outer view_label: "4 - Comms & Social Media Metrics" sql_on: ${fluentd_log.date_date} = ${v_communications.date_date} ;; relationship: many_to_one fields: [v_communications.post_author,v_communications.post_body...] } join: v_health_stats { type: left_outer view_label: "2 - Health Metrics" sql_on: ${fluentd_log.date_date} = ${v_health_stats.date_date} ;; relationship: many_to_one fields: [v_health_stats.total_steps, v_health_stats.total_distance_km...] } ``` ``` join: v_car_trips { type: left_outer view_label: "6 - Car Journey Metrics" sql_on: ${v_car_trips.date_date} = ${fluentd_log.date_date} ;; relationship: many_to_one fields: [v_car_trips.fuel_econ_kpl...n] } } ``` Truth be told, the net effect of all this isn’t conceptually much different to the enterprise semantic layers I’ve built for Oracle Business Intelligence and other similar systems over the years, and LookML is in many ways really just today’s implementation of what [nQuire did back in the late 90’s](https://www.rittmanmead.com/blog/2007/09/a-potted-history-of-oracle-bi-suite-enterprise-edition/) with Logical SQL and their logical and physical abstraction layers over physical data sources; but again, crucially, Looker’s LookML metadata layer was designed primarily for use through scripting and modern software development practices… ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2Ff9a21-2c61fffa-d764-40a9-9522-d9da40e93c28_800x536.png%3Fssl%3D1) and aligns perfectly with the development lifecycle and tooling used by the types of millenial developers who wouldn’t be seen dead working with the [sort of development lifecycle I used to present about](https://s3.amazonaws.com/rmc_docs/rmoug2012_rittman_obi_deployment.pdf) a few years ago for an earlier generation of BI tools. * * * What this gives me once all the development work is complete and data is ready to be analyzed in looks and dashboards is something that looks like the screenshot below; note the semantic model on the left-hand side with subject areas corresponding to the BigQuery views but with further grouping, friendly labelling and organizing into measures and dimension hierarchies and attributes. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2Fdc457-549f35c3-2164-47af-822a-84767aa1d43b_531x814.png%3Fssl%3D1) As Looker’s SQL queries the underlying BigQuery data store directly rather than an aggregated subset of that data, I can query right down to the lowest detailed-level events in the BigQuery dataset with new events streaming in all the time. Using Looker’s set of tabular, chart and mapping visualization options I can map out where I’ve been over period of time, what I did and correlate that with other data in the dataset, each view within the Looker explore joined together by date so it becomes one big dataset for analysis, as I tweeted about one evening last week when it all came together. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2F4f71a-1ec1dcde-88ee-443e-957b-c1a0645c03d7_800x487.jpeg%3Fssl%3D1) > More [@LookerData](https://twitter.com/lookerdata?ref_src=twsrc%5Etfw) fun with @GCPBigData [@smartthings](https://twitter.com/smartthings?ref_src=twsrc%5Etfw) and [@IFTTT](https://twitter.com/IFTTT?ref_src=twsrc%5Etfw), now mashing-up with my [@movesapp](https://twitter.com/movesapp?ref_src=twsrc%5Etfw) geolocation feed via LookML and mapping viz [pic.twitter.com/StKMpYfnnb](https://t.co/StKMpYfnnb) > > — Mark Rittman (@markrittman) [April 26, 2017](https://twitter.com/markrittman/status/857377608148897792?ref_src=twsrc%5Etfw) Now it’s all running there’s more I’d like to do with the various Google Cloud Platform APIs including using their new [Cloud Vision API](https://cloud.google.com/vision/) to classify and make searchable all the family photos I’ve collected over the years, and there’s lots of Looker features and capabilities I’ve not yet explored in detail including [Looker Actions](https://looker.com/news/press/data-actions), a concept very familiar from my days working with Oracle BI and its [Action Framework](https://docs.oracle.com/cd/E14571_01/bi.1111/e16364/actions.htm). For now thought I thought it would be interesting to share my thoughts on this new platform and the development process from the perspective of a long-term traditional BI+DW developer and architect, feel free to add comments or ask questions if you’re thinking of starting out with BigQuery, Looker or Fluentd in the future. --- # Slides from My “New World Hadoop Architectures (& What Problems They Really Solve) for Oracle DBAs”… Canonical: https://rittmananalytics.com/blog/2017/02/23/slides-from-my-new-world-hadoop-architectures-what-problems-they-really-solve-for-oracle-dbas URL: https://rittmananalytics.com/blog/2017/02/23/slides-from-my-new-world-hadoop-architectures-what-problems-they-really-solve-for-oracle-dbas Description: If you enjoyed my post the other day about BigQuery, Looker and Big Data’s Rediscovery of Data Warehousing … at Google Scale I’ve uploaded the slides on a talk I gave… Author: Mark Rittman Published: 2017-02-24T00:26:04+01:00 Modified: 2017-02-24T00:26:04+01:00 --- If you enjoyed my post the other day about [BigQuery, Looker and Big Data’s Rediscovery of Data Warehousing … at Google Scale](https://medium.com/mark-rittman/bigquery-looker-and-big-datas-rediscovery-of-data-warehousing-and-semantic-models-at-google-54f7c4ab6975?source=collection_home---6------0----------) I’ve uploaded the slides on a talk I gave today on this topic at the [UKOUG Database and RAC SIG event](http://www.ukoug.org/what-we-offer/news/new-world-hadoop-architectures/) in London. [https://www.slideshare.net/markrittman/new-world-hadoop-architectures-what-problems-they-really-solve-for-oracle-dbas](https://www.slideshare.net/markrittman/new-world-hadoop-architectures-what-problems-they-really-solve-for-oracle-dbas) The idea for the session came from a request from one of the SIG member to come in and talk about why these “new-world” Hadoop and NoSQL technologies were introduced over the years, what problem they solved at the time and how they solved it. The presentation goes through the various evolutions of data warehousing and big data technologies over the years since 2003 and finishes off with a look at the Google Cloud Platform and Oracle Elastic Compute Big Data platforms I’m using in the work I’m doing today. Thanks again to the UKOUG for hosting the event in London, and to Patrick Hurley and the SIG committees for inviting me to speak. --- # Slides from My “New World Hadoop Architectures (& What Problems They Really Solve) for Oracle DBAs”… Canonical: https://rittmananalytics.com/blog/2017/02/23/2017-02-23-2017-02-23-slides-from-my-new-world-hadoop-architectures-what-problems-they-really-solve-for-oracle-dbas URL: https://rittmananalytics.com/blog/2017/02/23/2017-02-23-2017-02-23-slides-from-my-new-world-hadoop-architectures-what-problems-they-really-solve-for-oracle-dbas Description: If you enjoyed my post the other day about BigQuery, Looker and Big Data’s Rediscovery of Data Warehousing … at Google Scale I’ve uploaded the slides on a talk I gave… Author: Mark Rittman Published: 2017-02-23T23:26:04+00:00 Modified: 2017-02-23T23:26:04+00:00 --- If you enjoyed my post the other day about [BigQuery, Looker and Big Data’s Rediscovery of Data Warehousing … at Google Scale](https://medium.com/mark-rittman/bigquery-looker-and-big-datas-rediscovery-of-data-warehousing-and-semantic-models-at-google-54f7c4ab6975?source=collection_home---6------0----------) I’ve uploaded the slides on a talk I gave today on this topic at the [UKOUG Database and RAC SIG event](http://www.ukoug.org/what-we-offer/news/new-world-hadoop-architectures/) in London. [https://www.slideshare.net/markrittman/new-world-hadoop-architectures-what-problems-they-really-solve-for-oracle-dbas](https://www.slideshare.net/markrittman/new-world-hadoop-architectures-what-problems-they-really-solve-for-oracle-dbas) The idea for the session came from a request from one of the SIG member to come in and talk about why these “new-world” Hadoop and NoSQL technologies were introduced over the years, what problem they solved at the time and how they solved it. The presentation goes through the various evolutions of data warehousing and big data technologies over the years since 2003 and finishes off with a look at the Google Cloud Platform and Oracle Elastic Compute Big Data platforms I’m using in the work I’m doing today. Thanks again to the UKOUG for hosting the event in London, and to Patrick Hurley and the SIG committees for inviting me to speak. --- # BigQuery, Looker and Big Data’s Rediscovery of Data Warehousing and Semantic Models … at Google… Canonical: https://rittmananalytics.com/blog/2017/02/21/2017-02-21-2017-02-21-bigquery-looker-and-big-datas-rediscovery-of-data-warehousing-and-semantic-models-at-google URL: https://rittmananalytics.com/blog/2017/02/21/2017-02-21-2017-02-21-bigquery-looker-and-big-datas-rediscovery-of-data-warehousing-and-semantic-models-at-google Description: Hadoop solved two problems data warehouse owners faced in the mid-2000’s; how to handle the explosion in data volumes brought on by the digitisition of business, and how to bring… Author: Mark Rittman Published: 2017-02-21T18:14:35+00:00 Modified: 2017-02-21T18:14:35+00:00 --- ![BigQuery, Looker and Big Data’s Rediscovery of Data Warehousing and Semantic Models … at Google…](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2F23f0e-a3e8af64-f0af-4fa2-82e7-f374fd713a0c_800x772.png%3Fssl%3D1) Hadoop solved two problems data warehouse owners faced in the mid-2000’s; how to handle the explosion in data volumes brought on by the digitisition of business, and how to bring agility and flexibility into the way data was landed, processed and made available to users. Traditional data warehousing database server technology got more expensive and unreliable the more nodes you added to a cluster whereas Hadoop became more resilient, and as open-source technology it was free to download and run on existing commodity servers. NoSQL databases emphasised speed over immediate consistency and offered storage options other than just tables and columns for applications that needed to load data in real-time and datatypes were increasingly unstructured and non-relational. The Hadoop and NoSQL ecosystem grew over the next ten years to include technologies such [Hive](https://cwiki.apache.org/confluence/display/Hive/Home), [Kafka](https://kafka.apache.org/) and [Sqoop](http://sqoop.apache.org/) that took concepts well-established in the database world and rethought them for the distributed, polyglot datasource and flexible-processing world of big data. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2F23f0e-a3e8af64-f0af-4fa2-82e7-f374fd713a0c_800x772.png%3Fssl%3D1) Most, like Hive, have areas of functionality clearly still years behind their commercial equivalent but focus instead on new possibilities opened-up by Hadoop’s flexible architecture and the ability of its open-source developer community to rapidly iterate and fill-in missing features as needs became apparent. Step-forward to two-or-three years ago and most large commercial organizations had big data initiatives of one form or another hosted in their data centres, using Hadoop distributions from the likes of Cloudera and Hortonworks on white-box hardware or dedicated server appliances from Oracle and IBM. In-house teams managed the often hundreds or thousands of nodes in a cluster, and over time [reference architectures](http://www.oracle.com/ocom/groups/public/@otn/documents/webcontent/2297765.pdf) emerged that added governance and workload-separation concepts demanded by enterprise customers, along with [design patterns](https://en.wikipedia.org/wiki/Lambda_architecture) that bridged the gap between fast-moving streaming data sources and the slower, batch-oriented world of traditional on-premise data-processing. And so Hadoop and NoSQL technologies [increasingly became the default choice organizations made when storing and organizing data for query and analysis](http://blogs.barrons.com/techtraderdaily/2015/03/30/michael-stonebraker-describes-oracles-obsolescence-facebooks-enormous-challenge/) as they were cheaper, more flexible in-terms of how and when you processed data and were orders of magnitude cheaper to license and provision than the high-end relational databases servers they’d used up until. [I’m giving at talk on this topic](http://www.ukoug.org/what-we-offer/news/new-world-hadoop-architectures/) later this week at the UK Oracle User Group’s Database SIG in London, and the problems each Hadoop component solved and the workloads that still make sense to run on traditional relational database technologies are key concepts DBAs and developers need to be aware of when thinking about these new technologies. But as with any new technology that makes an older one obsolete Hadoop and NoSQL introduced their own new problems, and as we discussed in the [Drill to Detail Podcast episode featuring colleague Alex Olivier](https://www.drilltodetail.com/drill-to-detail/2017/1/24/drill-to-detail-ep16-qubit-visitor-cloud-google-bigquery-with-special-guest-alex-olivier) in this case they were around the complexity and effort required to manage clusters containing [thousands of nodes](http://www.forbes.com/sites/danwoods/2014/06/26/how-qubits-data-supply-chain-shows-the-power-of-hbase/#590e42a257b7) running dozens of services doing so economically and securely, and in addition the cognitive overhead put on users by Hadoop and NoSQL’s schema-on-read storage meant pushing the burden of structuring and interpreting data onto users who didn’t have the skills or time do this effectively. The first generation of cloud platforms for running big data workloads typically took what was done on-premise and lifted-and-shifted that workload into the cloud replacing physical servers with virtual ones, giving system owners more capacity and flexibility around server provisioning but still requiring the customer to own and manage the environment themselves. The first cloud database server platforms were most likely to be straight ports of that vendor’s on-premise product hosted in VMs running on that cloud vendor’s Infrastructure-as-a-Service (Iaas) platform, but a new generation of elastic, cloud-native data warehouse Platform-as-a-Service products such as [Google BigQuer](https://cloud.google.com/bigquery/)y and [Amazon Athena](https://aws.amazon.com/athena/) bring together the benefits of cloud and big data scale, come with their own similarly elastically-provisioned [data processing](https://cloud.google.com/dataflow/) and [data ingestion services](https://cloud.google.com/pubsub/) while re-introducing concepts such as tabular storage and SQL access and On-premise technologies such as [Apache Kudu](https://kudu.apache.org/) started this move towards structuring the storage big data systems used for analytic workloads as tables and columns rather than schema-less filesystems, but BigQuery took this mainstream to the point where organizations such as [Qubit](http://www.qubit.com/), the London startup I featured on the Drill to Detail episode and where I’m currently responsible for their analytics product strategy, use it to [ingest, process and stored hundreds of terabytes of customer event data each day](https://www.youtube.com/watch?v=9nhoWpPKA1M&feature=youtu.be) and store it neatly organized into tables, columns, projects and industry-specific business views. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2F93ed9-bdc3ed5b-4062-4fd8-93ad-26767720418b_800x752.png%3Fssl%3D1) If you logged into BigQuery’s SQL web interface and didn’t know about the massively-parallel big data architecture BigQuery took from Dremel you’d assume it was just another database server, and more importantly you can get on with your analysis rather than struggling with JSON SerDes and Hive EXPLODE and LATERAL VIEW clauses to decode your schema-on-read dataset. In today’s service-based PaaS cloud analytics platforms the two previously distinct relational data warehousing and Hadoop distributed processing technologies are [converging with cloud storage and elastically-provisioned compute and storage](https://cloud.google.com/solutions/bigquery-data-warehouse) to give us something new and potentially very interesting … just as new BI tool startup Looker coming out of the big data and e-commerce space has begun introducing a new generation of software developers and data architects to the benefits of BI data modeling, semantic models and data-layer abstraction. The trouble with using tools such as Tableau and Oracle Data Vizualization Desktop against even tabular-structured data sources like Google BigQuery for business-user analysis is that they treat each report as it’s own silo of data; users can’t easily navigate from one area of data to another as reports are based on single data extracts of metrics and attributes from one subject area designed to make data visualizations and data discovery simple for departmental users. Traditional enterprise BI tools like Business Objects and Cognos do give users the ability to create rich, integrated semantic business models over platforms such Google BigQuery using point-and-click graphical tools, but in the kinds of startups and engineering-led early-adopter companies implementing BigQuery and wanting to structure and join-together reporting data as create enterprise semantic models they’re more interested in working with markup languages, Git repositories and doing it all in code. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2F67b96-ae8d5d27-812e-4d2e-9aa9-f97aa796f65f_800x483.png%3Fssl%3D1) [LookML](https://looker.com/product/lookml), a language developed by Looker for describing dimensions, aggregates, calculations and data relationships in a SQL database is used to define semantic models in Looker’s BI tool and is aimed squarely at the software engineers typically working on the e-commerce and big data platforms that use BigQuery to host and analyze big data customer activity datasets. The code snippet below shows some simple LookML used to define an orders view in a Looker semantic model, and developers are encouraged to share code examples on Looker’s community [Looker Blocks Director](https://looker.com/looker-blocks/directory)y. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2F77492-109d303e-67a1-4edd-9a8b-79994894d137_593x606.png%3Fssl%3D1) We’re using Looker at Qubit to create semantic models for use with [Live Tap](http://www.qubit.com/solutions/visitor-cloud-data-exchange), an analytics service that runs on top of Google BigQuery that marketers use to analyze and explore the event-level customer activity datasets we build for them. Although BigQuery provides SQL access and a web UI for running queries we knew marketers would prefer to use a graphical BI tool, and Looker and the sample code we’re planning on sharing on Github for customers to use and extend in their own Looker environments means we can present all of their data joined-up, with meaningful business names for metrics and dimension attributes and organized into subject areas that hide the complexity of the underlying BigQuery table structures. And so the story ends up going full loop; big data solves the scale and agility issues that were holding back data warehouses, but added complexity that in-time was addressed by running that infrastructure in the cloud as a serviceeventually was addressed by running it as a service in the cloud, and now in-turn adopts some of the characteristics of the platforms it usurped to make developers more productive and help end-users understand and analyze all their corporate data as one big joined-up dataset. [Google Cloud’s Next developer event](https://cloudnext.withgoogle.com/) is running in San Francisco in a couple of week’s time and it’ll be interesting to see what’s coming next for BigQuery, and in another case of big data adopting traditional database capabilities I’ll definately be interested in hearing about Google Cloud Spanner — and how it could do for OLTP what MapReduce and Dremel did for our data warehousing relational databases. --- # Graph Analysis in the “How a Tweet Went Viral” Conference Presentation Canonical: https://rittmananalytics.com/blog/2017/02/03/graph-analysis-in-the-how-a-tweet-went-viral-conference-presentation URL: https://rittmananalytics.com/blog/2017/02/03/graph-analysis-in-the-how-a-tweet-went-viral-conference-presentation Description: Earlier this week I presented a session at the BIWA 2017 conference in San Francisco on using Oracle Big Data Spatial & Graph to understand how my WiFi kettle tweet… Author: Mark Rittman Published: 2017-02-03T18:48:11+01:00 Modified: 2017-02-03T18:48:11+01:00 --- Earlier this week I presented a session at the [BIWA 2017 conference in San Francisco](http://www.biwasummit.org/speakers/mark-rittman/) on using Oracle Big Data Spatial & Graph to understand how my [WiFi kettle tweet went viral](https://medium.com/mark-rittman/the-story-behind-the-ikettle-the-eleven-hour-struggle-to-make-a-cup-of-tea-and-why-it-was-all-769144d12d7) back in October last year, by using graph analysis and data visualization tools like Cytoscape and [Tom Sawyer Perspectives](https://www.tomsawyer.com/products/perspectives/). [https://www.slideshare.net/markrittman/how-a-tweet-went-viral-biwa-summit-2017](https://www.slideshare.net/markrittman/how-a-tweet-went-viral-biwa-summit-2017) I’ve [uploaded the slides](http://www.slideshare.net/markrittman/how-a-tweet-went-viral-biwa-summit-2017) and embedded the demos as a series of Youtube videos within the slides, so you can see the new Timeslice-analysis feature in the Oracle Cytoscape plugin we developed for the presentation along with the mapping and analysis features in Tom Sawyer Perspectives that helped us work out exactly how the story went viral and who helped this happen … as you’ll find out in the slides. --- # Qubit’s Journey to Petabyte-Scale Machine-Learning and Analytics on Google Cloud Platform … and… Canonical: https://rittmananalytics.com/blog/2017/01/24/qubits-journey-to-petabyte-scale-machine-learning-and-analytics-on-google-cloud-platform-and URL: https://rittmananalytics.com/blog/2017/01/24/qubits-journey-to-petabyte-scale-machine-learning-and-analytics-on-google-cloud-platform-and Description: My guest on this week’s Drill to Detail Podcast is Alex Olivier from Qubit, a startup based in London founded by four ex-Googlers that uses big data technology and machine… Author: Mark Rittman Published: 2017-01-24T14:20:32+01:00 Modified: 2017-01-24T14:20:32+01:00 --- ![Qubit’s Journey to Petabyte-Scale Machine-Learning and Analytics on Google Cloud Platform … and…](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2017%2F01%2F1752277798426-95c02q.png%3Ffit%3D800%252C291%26ssl%3D1) My guest on [this week’s Drill to Detail Podcast](https://www.drilltodetail.com/podcast/2017/1/24/drill-to-detail-ep16-qubit-visitor-cloud-google-bigquery-with-special-guest-alex-olivier) is Alex Olivier from [Qubit](http://www.qubit.com/), a startup based in London founded by four ex-Googlers that [uses big data technology and machine learning](https://techcrunch.com/2014/09/29/qubit-series-b/) to deliver personalized experiences and product recommendations to customers of some of the biggest names in e-commerce, travel and online gaming. Qubit’s innovation in the space was to move beyond simple A:B testing and cookie-based personalization to create an [event-level, petabyte-scale customer activity data lake](http://www.qubit.com/our-infrastructure) running in Google Cloud that enables retailers and other organizations deliver personalize offers and site features based on a much more granular understanding of customer behavior and preferences, the same vision in fact that I kept putting forward in my webinars and presentations last year around [data reservoirs and customer 360-degree analysis](https://speakerdeck.com/markrittman/adding-a-data-reservoir-to-your-oracle-data-warehouse-for-customer-360-degree-analysis). ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fblog-images%2F1752277798426-95c02q.png%3Fssl%3D1) And of course, this centralized, event-level store of customer activity and purchase preferences creates a fantastic platform on which to build predictive models, real-time next-best-offer decision engines… ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fblog-images%2F1752277798694-uw61jr.png%3Fssl%3D1) … and enable real-time big data analytics — with Qubit’s product in this area, [Live Tap](http://www.qubit.com/solutions/analytics), being what I’ve been working on since last year’s Openworld advising their Product Management team and working under Paul Rodwick, who some of you might know from his time as head of Oracle’s BI Product Development team. I’ll write about Live Tap and the work I’m doing there, and my experiences creating an analytics product on top of BigQuery at a later date, as well as our use of [Looker](https://looker.com/) to create a semantic model over Qubit’s event-level data lake. The podcast episode with Alex Olivier talks about Qubit’s journey from initially using Amazon AWS to land and process data using S3 buckets and MapReduce, then moving it all onsite to a [cluster of thousands of HBase region servers](http://www.forbes.com/sites/danwoods/2014/06/26/how-qubits-data-supply-chain-shows-the-power-of-hbase/#2551576157b7) storing data ingested and processed using Storm with latency down to four hours, to their current setup using Google BigQuery, Google PubSub and Google Cloud Dataflow processing 100,000 events per second and making it all available to customers with latency around 5 seconds — seriously impressive and a great case study around the use of cloud-hosted, elastically-provisioned big data analytics platforms. There’s also a video of [Alex presenting on Qubit’s architecture at last year’s Google NEXT event](https://youtu.be/9nhoWpPKA1M) in London on Youtube, and this episode along with all the others is also available for download on the [iTunes Podcast Directory](https://itunes.apple.com/gb/podcast/drill-to-detail-podcast/id1141126960?mt=2&ign-mpt=uo%3D4) where you can subscribe for free and automatically download new episodes as they become available Tuesday of every week. --- # Drill to Detail Podcast : Looking Back at 2016, and What’s New and Planned for 2017 Canonical: https://rittmananalytics.com/blog/2016/12/31/drill-to-detail-podcast-looking-back-at-2016-and-whats-new-and-planned-for-2017 URL: https://rittmananalytics.com/blog/2016/12/31/drill-to-detail-podcast-looking-back-at-2016-and-whats-new-and-planned-for-2017 Description: I started the Drill to Detail Podcast series back in October this year with an inaugural episode featuring long-time friend and colleague Stewart Bryson talking about changes in the BI… Author: Mark Rittman Published: 2016-12-31T02:17:39+01:00 Modified: 2016-12-31T02:17:39+01:00 --- ![Drill to Detail Podcast : Looking Back at 2016, and What’s New and Planned for 2017](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2016%2F12%2F1752277798994-8uv83q.png%3Ffit%3D800%252C563%26ssl%3D1) I [started the Drill to Detail Podcast series](http://www.drilltodetail.com/drill-to-detail/2016/12/29/test-post) back in October this year with an [inaugural episode featuring long-time friend and colleague Stewart Bryson](http://www.drilltodetail.com/podcast/2016/9/11/drill-to-detail-ep1-after-the-gartner-bia-magic-quadrant-2016-with-special-guest-stewart-bryson) talking about changes in the BI industry, and since then we’ve gone on to publish one episode a week featuring guests such as [Oracle’s Paul Sonderegger](http://www.drilltodetail.com/podcast/2016/10/23/rsmn3fbumx1p7zew7kbopykm5x6qa2) on data capital, [Cloudera’s Mike Percy](http://www.drilltodetail.com/podcast/2016/9/14/4q91yzszw10529f83veq9l6jo29091) on Apache Kudu followed shortly after by [Mark Grover](http://www.drilltodetail.com/podcast/2016/11/1/drill-to-detail-ep7-apache-spark-and-hadoop-application-architectures-with-special-guest-mark-grover) on Apache Spark, [Dan McClary from Google](http://www.drilltodetail.com/podcast/podcast/2016/8/4/drill-to-detail-ep2-future-of-sql-on-hadoop-with-special-guest-dan-mcclary) talking about SQL-on-Hadoop and [Neeraja Rentachintala from MapR](http://www.drilltodetail.com/podcast/2016/12/13/drill-to-detail-ep13-apache-drill-mapr-and-data-discovery-with-special-guest-neeraja-rentachinta) telling us about Apache Drill and Apache Arrow, [Tanel Poder from Gluent](http://www.drilltodetail.com/podcast/2016/12/6/drill-to-detail-ep12-gluent-and-the-new-world-of-hybrid-data-with-special-guest-tanel-poder) on the New World of Hybrid Data … ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fblog-images%2F1752277798994-8uv83q.png%3Fssl%3D1) … along with many other guests including [Jen Underwood](http://www.drilltodetail.com/podcast/2016/11/8/drill-to-detail-ep8-self-service-bi-data-prep-big-data-vendor-strategy-with-special-guest-jen-underwood), [Kent Graziano](http://www.drilltodetail.com/podcast/2016/9/14/yffvlkgzk1099zpz944v4ttfd9njp1), [Pat Patterson](http://www.drilltodetail.com/podcast/2016/11/15/drill-to-detail-ep9-streamsets-data-in-motion-and-data-drift-with-special-guest-pat-patterson) from StreamSets and old friends and colleagues [Andrew Bond](http://www.drilltodetail.com/podcast/2016/9/14/6srrkbbeboig0s8r4w5tpj38wwc0ds), [Graham Spicer](http://www.drilltodetail.com/podcast/2016/11/29/drill-to-detail-ep11-olap-consulting-bi-past-present-future-with-special-guest-graham-spicer) and most recently for the Christmas and New Year special, [Robin Moffatt](http://www.drilltodetail.com/podcast/2016/12/20/drill-to-detail-ep14-christmas-new-year-special-with-special-guest-robin-moffatt). In fact I’d only ever planned on publishing new episodes of Drill to Detail once every two weeks along the lines of the two podcasts that inspired Drill to Detail, the Apple-focused *[The Talk Show](https://daringfireball.net/thetalkshow/)* by John Gruber and Marco Arment, Casey Liss, and John Siracusa’s *[Accidental Tech Podcast](http://atp.fm/)*), but what with a number of episodes recorded over the summer waiting for the October launch and so many great guests coming on over the new few months we ended-up publishing new episodes every week. So at the end of 2016 and with fourteen episodes [published on this website](http://www.drilltodetail.com/) and on the [iTunes directory](https://itunes.apple.com/gb/podcast/drill-to-detail/id1141126960?mt=2) I’d like to take this opportunity to thank all the guests that came on the show along with friends in the industry such as [Confluent’s Gwen Shapira](https://www.confluent.io/blog/author/gwen-shapira/) who helped get the word out and make introductions, and of course most importantly I’d like to thank everyone who’s downloaded episodes of the show, mentioned it on Twitter and other social networks and increasingly, subscribed to the show on the iTunes store to the point where we’re typically hitting a thousand or more subscribers each week based on Squarespace’s estimate of overall RSS subscriber numbers including those coming in from iTunes and other feed aggregators. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fblog-images%2F1752277799271-kso0r.png%3Fssl%3D1) And if you’re wondering which show had the highest audience numbers it was [November’s Episode 7](http://www.drilltodetail.com/podcast/2016/11/1/drill-to-detail-ep7-apache-spark-and-hadoop-application-architectures-with-special-guest-mark-grover) with Cloudera’s Mark Grover on Apache Spark and Hadoop Application Architectures, closely followed by [October’s episode](http://www.drilltodetail.com/podcast/2016/10/23/rsmn3fbumx1p7zew7kbopykm5x6qa2) with Oracle’s Big Data Strategist Paul Sonderegger on data capital, both of which were great examples of what ended-up being the recurring theme and area of discussion with every one of the guests and shows we recorded … the *business, strategy, rationale and opportunities for competitive advantage* coming out of innovations in the big data and analytics industry. * * * And now we’re going into 2017 and the second year of Drill to Detail, we’re going to double-down on this area of focus by updating the Drill to Detail website with a new look and launching the new [Drill to Detail Blog](http://www.drilltodetail.com/drill-to-detail/) to accompany the podcast series, each week posting a long-form blog post looking at the business and strategy behind what’s happening in the big data, analytics and cloud-based data processing industry. We’ll still be continuing with the podcast series exactly as they are now with guests including Elastic’s Mark Walkom and Cindi Howson from Gartner due on the show in January, but these longer-form blog posts give us a chance to explore and explain in a more structured way the topics and questions raised by what’s been discussed on the podcast, analyzing and exploring the implications from trends and directions coming out of the industry. Finally, going back to my original inspiration for the podcast that started all of this, a big part of the inspiration and idea to focus on this particular theme came from what’s now become my new favourite blog and podcast series, Ben Thompson’s [Stratechery](https://stratechery.com/) website and [Exponent](http://exponent.fm/) podcast that he co-authors with James Allworth, and if I manage to get even someway towards the insights and understanding he brings towards the wider IT landscape and apply that to the part of the industry we work in during the coming year … well that’ll be my evenings, commute time and weekend time well spent this coming year. --- # Data Lakes at Google Scale, The End of Meaningless Customer Experiences, and UKOUG Tech’16 in… Canonical: https://rittmananalytics.com/blog/2016/12/02/data-lakes-at-google-scale-the-end-of-meaningless-customer-experiences-and-ukoug-tech16-in URL: https://rittmananalytics.com/blog/2016/12/02/data-lakes-at-google-scale-the-end-of-meaningless-customer-experiences-and-ukoug-tech16-in Description: I’m speaking at [#ukoug\_tech16](https://twitter.com/hashtag/ukoug_tech16?src=hash&ref_src=twsrc%5Etfw) SuperSunday and Monday on Big Data, IoT, Analytics … and “Have I Got News For You” [https://t.co/bwx7U7sKIu](https://t.co/bwx7U7sKIu) [pic.twitter.com/PaLurCGnIR](https://t.co/PaLurCGnIR) > > — Mark Rittman (@markrittman) [December 1, 2016](https://twitter.com/markrittman/status/804444430967119872?ref_src=twsrc%5Etfw) … just because I spent all day trying to voice-control my kettle and feed its data into the Hadoop cluster in my garage, as you do. See some of you in Birmingham next week, and in Malta for another Oracle event if anyone’s there as well. --- # Google BigQuery, and Why Big Data is About to Have its GMail Moment Canonical: https://rittmananalytics.com/blog/2016/11/19/google-bigquery-and-why-big-data-is-about-to-have-its-gmail-moment URL: https://rittmananalytics.com/blog/2016/11/19/google-bigquery-and-why-big-data-is-about-to-have-its-gmail-moment Description: If like me you ran your own email server back in the 90’s and spend evenings and weekends keeping it running and wondering what all the fuss was about open… Author: Mark Rittman Published: 2016-11-19T21:15:51+01:00 Modified: 2016-11-19T21:15:51+01:00 --- ![Google BigQuery, and Why Big Data is About to Have its GMail Moment](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2016%2F11%2F1752277799787-8vqwyq.jpeg%3Ffit%3D800%252C446%26ssl%3D1) If like me you ran your own email server back in the 90’s and spend evenings and weekends keeping it running and wondering what all the fuss was about open relays, or you remember using Outlook around that same time and working with tiny mailbox quotas and seemingly constant Exchange Server outages, you’ll probably remember how revolutionary [Google’s GMail service seemed in 2004](https://en.m.wikipedia.org/wiki/History_of_Gmail) with more-or-less unlimited storage, one of the first AJAX JavaScript rich web clients and all of the backend infrastructure taken care of by Google. Fast-forward to today and it’s increasingly common to see what’s now Google Apps deployed across large corporations, universities and even government deparments as they switch to buying email, calendering and file storage as services rather than own and manage the infrastructure themselves, and users benefit by having a more reliable service that always has enough capacity to meet their needs. Contrast this with the state of Hadoop and on-premise big data systems today, where its not unusual to wait months for corporate IT to add a new cluster or expand capacity on existing ones as they in turn wait for procurement to negotiate a deal and then pass the work to an outsourced managed service provider to finally have it provisioned in the data centre – if there’s still some CapEx budget left, otherwise try again next year. Hadoop system administration is still a bit of a black art and few of us have the skills and experience to manage and keep secure clusters of thousands or even hundreds of server nodes … but that exactly what the likes of Google, Microsoft, Amazon Web Services and Oracle do for a living, and [as I blogged about a few months ago](https://medium.com/mark-rittman/oracles-big-data-platform-goes-cloud-becomes-elastic-and-suddenly-looks-very-interesting-dd99f1cf6d98?source=collection_home_________5_________---6------5) they’re now selling object-level storage, Spark and streaming ingestion as services you connect to and consume, billed monthly based on numbers of users or some other metric and never having worry about upgrades, cluster administration or replacing a faulty node. > Final state of my home automation + wearables datalake project as I finish sabbatical – now ported to GCS, Google BigQuery & Data Studio [pic.twitter.com/pTNjvImHcU](https://t.co/pTNjvImHcU) > > — Mark Rittman (@markrittman) [November 4, 2016](https://twitter.com/markrittman/status/794332604346548224?ref_src=twsrc%5Etfw) And in the same way I worked out that my time was better spent learning Oracle development rather than running a mail server, the Hadoop cluster I’ve been tinkering with in the garage is likely to go the same way as I [port my home development work over to Google’s cloud platform](https://twitter.com/markrittman/status/794332604346548224) and think of my big data platform as elastic compute and storage services that always work and scale-up when needed … and as big data and data warehousing platforms converge as they transform into managed cloud services, big data doesn’t necessarily have to be Hadoop, HDFS and unstructured data stores. * * * Google famously invented the core Google File System and MapReduce technologies that Doug Cutting and Mike Cafarella then reimplemented as the open source Apache Hadoop project, but Google then went on to create [Dremel, a distributed ad-hoc query system](http://research.google.com/pubs/pub36632.html) better suited to data warehouse-type workloads than batch-orientated MapReduce, ran over tens of thousands of servers in Google’s data centres and stored its data in column-oriented format (and in-turn inspired another open-source project, Apache Drill) ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fblog-images%2F1752277799787-8vqwyq.jpeg%3Fssl%3D1) Google have now made a public version of Dremel, [Google BigQuery](https://cloud.google.com/bigquery/), available as a service within Google Cloud Platform and conceptually it’s similar to the Apache Kudu and Apache Impala projects that Mike Percy and I discussed in [Episode 3 of the Drill to Detail podcast](https://medium.com/mark-rittman/drill-to-detail-ep-4210ff9ef4a0?source=collection_home_________4_________---6------4); specialised storage optimised for analytic workloads with data stored in a tabular format and a SQL query engine designed to work with BI tools that support BigQuery as a data source through its API and distinct SQL dialect, or more recently a JDBC interface and ANSI SQL. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fblog-images%2F1752277800031-5s48h.png%3Fssl%3D1) As I showed in my tweet about BigQuery I’ve been using it and its ability to use Google Sheets as external table sources to land and query my home IoT and wearables data, gradually building in real-time ingestion and loading using Google PubSub and Google Cloud DataFlow, to receive and then transform incoming IoT events in a pipeline that feeds data into BigQuery as streaming row inserts. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fblog-images%2F1752277800281-yjbtu.png%3Fssl%3D1) What’s interesting is that Google Cloud Platform and Google BigQuery is the technology that ended-up powering the type of [Customer 360 applications I talked about around this time last year](https://m.youtube.com/watch?list=PL6Vx6k9b2zbXptIN5777FNet6aephlSfe&v=Uux1CuAVdCs). ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fblog-images%2F1752277800571-forr1i.jpeg%3Fssl%3D1) While most Customer 360 and digital marketing initiatives start on in-house Hadoop clusters or cloud-based services such as Amazon Redshift, the petabyte-scale of event-based customer interaction records means that it’s easier, cheaper and far less work to hand this sort of workload off to Google and have developers concentrate on delivering new experiences and offers to customers rather than plugging in another truckload of servers into the cluster to try and keep up with demand. But the story behind that is one for another day … and it’s good. --- # Data Capital, Competitive Strategy and the Economics of Big Data — Drill to Detail Podcast Ep.6 Canonical: https://rittmananalytics.com/blog/2016/10/30/data-capital-competitive-strategy-and-the-economics-of-big-data-drill-to-detail-podcast-ep-6 URL: https://rittmananalytics.com/blog/2016/10/30/data-capital-competitive-strategy-and-the-economics-of-big-data-drill-to-detail-podcast-ep-6 Description: My work is done. And now onto everything else I meant to do today, after that first cup of tea. [pic.twitter.com/bJPuJ85TCT](https://t.co/bJPuJ85TCT) > > — Mark Rittman (@markrittman) [October 11, 2016](https://twitter.com/markrittman/status/785981145007714304?ref_src=twsrc%5Etfw) [https://platform.twitter.com/widgets.js](https://platform.twitter.com/widgets.js) … then wake up to find it’s gone viral on the internet, the phone is ringing off the hook, and you’ve now added the amount of cups of tea you drink each day to the dataset you’re analyzing back on your Hadoop cluster, along with around 1,500 internet comments about the story you’ve been telling and a similar number on the Guardian and other websites. All of which, though, will at least give me some interesting anecdotes and datasets to use at the [UK Oracle User Group Tech’16 event in Birmingham](http://tech16.ukoug.org/) later on in the year when I’m presenting on this topic — or more likely, how to get an iKettle working -in one of my conference sessions on big data, cloud and IoT analytics. For now though, that’s the story behind the iKettle, and why I still think it was worth waiting eleven hours (or even more, in the end) to get a voice-activated cup of tea and that essential start to the day … even though as it was 1am, it was actually now time to go bed. --- # Drill to Detail Ep.3 with Mike Percy on Apache Kudu Canonical: https://rittmananalytics.com/blog/2016/10/05/drill-to-detail-ep-3-with-mike-percy-on-apache-kudu URL: https://rittmananalytics.com/blog/2016/10/05/drill-to-detail-ep-3-with-mike-percy-on-apache-kudu Description: I doubt if these two things are unrelated (h/t [@ronenschwartz](https://twitter.com/ronenschwartz?ref_src=twsrc%5Etfw)) [https://t.co/LDd59mI40w](https://t.co/LDd59mI40w) [pic.twitter.com/KgIO8txDxT](https://t.co/KgIO8txDxT) > > — Mark Rittman (@markrittman) [September 28, 2016](https://twitter.com/markrittman/status/781147091921952768?ref_src=twsrc%5Etfw) As I said, offering Hadoop as services on an elastic cloud platform is nothing new, just tables stakes if Oracle wants to be in this game (and they do). Where things then got interesting is when the demo and the tooling came out. First off was a packaged industry IoT application using predictive analytics to tell the customer which device or sensor is likely to fail before it actually does, giving them the chance to repair or replace the component before there’s a service outage. So far, standard stuff though the execution and UX looked slick, something Oracle are increasingly good at if what you last saw was Oracle Forms and Java applets in the browser. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fblog-images%2F1752277802156-q2ugjl.png%3Fssl%3D1) Then we got on to the underlying services that provide data and insights for the IoT packaged app. You’ve probably seen articles and blogs I’ve written on Oracle’s Big Data Preparation Cloud Service tool, but Data Flow Machine Learning is Oracle’s long-anticipated cloud data integration service that uses dataflow-style mappings, Spark as the execution engine and machine-learning smarts and semantic discovery to automate as much of the development process as possible — [this paper back from Spark Summit 2015](https://spark-summit.org/2015/events/extending-word2vec-for-performance-and-semi-supervised-learning/) explains some of the supervised and semi-supervised machine learning algorithms that Oracle (via their Silver Creek Systems acquisition) used when creating their cloud data prep service. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fblog-images%2F1752277802699-z2omvc.png%3Fssl%3D1) Then the demo moved on to setting up data ingestion services using Kafka and Spark services running real-time and streaming, and then showed target schemas being defined and created automatically for the user. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fblog-images%2F1752277802984-6m653.png%3Fssl%3D1) And then it got on to Data Flow Machine Learning… ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fblog-images%2F1752277803348-x7hcz.png%3Fssl%3D1) … and that’s worthy of a blog post all of it’s own given how significant it’s going to be going forward, and I’ll post that in a day or so’s time. --- # Drill to Detail Ep.2. “The Future of SQL-on-Hadoop” with Special Guest Dan McClary Canonical: https://rittmananalytics.com/blog/2016/09/27/drill-to-detail-ep-2-the-future-of-sql-on-hadoop-with-special-guest-dan-mcclary URL: https://rittmananalytics.com/blog/2016/09/27/drill-to-detail-ep-2-the-future-of-sql-on-hadoop-with-special-guest-dan-mcclary Description: This is a good one. Most of you will know Dan McClary as the product manager at Oracle for Big Data SQL, and more recently he’s now moved to Google… Author: Mark Rittman Published: 2016-09-27T22:01:32+01:00 Modified: 2016-09-27T22:01:32+01:00 --- ![Drill to Detail Ep.2. “The Future of SQL-on-Hadoop” with Special Guest Dan McClary](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2016%2F09%2F1752277803679-ll61lb.png%3Ffit%3D219%252C219%26ssl%3D1) This is a good one. Most of you will know Dan McClary as the product manager at Oracle for Big Data SQL, and more recently he’s now moved to Google to work on their storage and big data projects. If you’ve met Dan or [heard him speak](https://www.youtube.com/watch?v=07Wm_uUoWqU) you’ll know he’s not only super-smart and very knowledgable about Hadoop, but he’s great to get into a conversation with .. which is why I was particularly pleased to have him on as the special guest on the [Episode 2](http://www.drilltodetail.com/podcast/podcast/2016/8/4/drill-to-detail-ep2-future-of-sql-on-hadoop-with-special-guest-dan-mcclary) of my new podcast series, [Drill to Detail](http://www.drilltodetail.com/). ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fblog-images%2F1752277803679-ll61lb.png%3Fssl%3D1) In this new episode Dan and I discuss the state of the SQL-on-Hadoop market and where he’s seeing the innovation; how the mega-vendors are contributing to, extending and competing with the Hadoop ecosystem; and what he’s seeing coming out of the likes of Google, Yahoo and other Hadoop innovators that may well make its way into the next Apache Hadoop project. You can [download the podcast recording](http://www.drilltodetail.com/podcast/podcast/2016/8/4/drill-to-detail-ep2-future-of-sql-on-hadoop-with-special-guest-dan-mcclary) from the Drill to Detail website, and it’s [just about to go live on iTunes](https://itunes.apple.com/gb/podcast/drill-to-detail-podcast/id1141126960?mt=2) where you can subscribe and automatically receive future episodes — and you certainly won’t want to miss the next one, believe me. --- # New Oracle Magazine Article on Oracle Big Data Spatial & Graph for Social Network Analysis Canonical: https://rittmananalytics.com/blog/2016/09/25/new-oracle-magazine-article-on-oracle-big-data-spatial-graph-for-social-network-analysis URL: https://rittmananalytics.com/blog/2016/09/25/new-oracle-magazine-article-on-oracle-big-data-spatial-graph-for-social-network-analysis Description: Back at the beginning of 2016 I presented a session at the BIWA 2016 Conference on using Oracle’s (at that time) new Big Data Spatial and Graph product to do… Author: Mark Rittman Published: 2016-09-25T07:45:14+01:00 Modified: 2016-09-25T07:45:14+01:00 --- ![New Oracle Magazine Article on Oracle Big Data Spatial & Graph for Social Network Analysis](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2016%2F09%2F1752277803921-8v8lug.gif%3Ffit%3D270%252C210%26ssl%3D1) ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fblog-images%2F1752277803921-8v8lug.gif%3Fssl%3D1) Back at the beginning of 2016 I [presented a session at the BIWA 2016 Conference](http://www.slideshare.net/rittmanmead/oracle-big-data-spatial-graph-social-media-analysis-case-study) on using Oracle’s (at that time) new Big Data Spatial and Graph product to do what’s called “social network analysis”; a form of graph analysis using the networks formed by social networks, in this case Twitter, to help understand who’s central to a particular network, who influences who, who are the best connectors through the network to someone you might want to speak to, and so on. Twitter works well as a network to analyse for these types of presentations as most people are familiar with the tweets, replies, hashtags, retweets and other types of connections users can create with this social network, and [Oracle Big Data Spatial and Graph](http://www.oracle.com/technetwork/database/database-technologies/bigdata-spatialandgraph/overview/index.html) helps you identify the communities that users not only self-declare through hashtags (for example, “[#00w16](https://twitter.com/hashtag/oow16?src=hash)”) but also through the connections they build up by just interacting with the same people over and over again — a graph analysis technique called [“clustering”](https://blogs.oracle.com/bigdataspatialgraph/entry/detecting_communities_in_a_social) that identifies groups of users based purely on their common networks of communication. If you’re interested in the topic and want to understand a bit more about how this type of graph analysis works, as well as see how this new Oracle Big Data product looks, the new [September/October 2016 edition](http://www.oracle.com/technetwork/oramag/magazine/home/index.html) of Oracle Magazine has a follow-up article by myself on the topic called [“Social Network Analysis : Use Oracle Big Data Spatial and Graph to analyze social networks”](http://www.oracle.com/technetwork/issue-archive/2016/16-sep/o56ba-3211403.html) that uses some Twitter data I provide along with [software you can download](http://www.oracle.com/technetwork/database/database-technologies/bigdata-spatialandgraph/downloads/index.html) from OTN if you want to try it out yourself. Once you’ve worked through the article, be sure to check out the official blog from the product team for more scripts and examples including [product recommendations](https://blogs.oracle.com/bigdataspatialgraph/entry/intuitive_explanation_of_personalized_page) using the personalized page rank algorithm, and [fraud detection in finance](https://blogs.oracle.com/bigdataspatialgraph/entry/identifying_potential_fraud_activities_in) spotted by identifying circular payment relationships — two very good examples of data analysis that would be very hard to perform using traditional relational data models. --- # OBIEE12c — Pushing-Up Daisies … or more Relevant than Ever in the World of BiModal IT? Canonical: https://rittmananalytics.com/blog/2016/09/25/obiee12c-pushing-up-daisies-or-more-relevant-than-ever-in-the-world-of-bimodal-it URL: https://rittmananalytics.com/blog/2016/09/25/obiee12c-pushing-up-daisies-or-more-relevant-than-ever-in-the-world-of-bimodal-it Description: Tomorrow I’m going back to my roots and presenting on Oracle BI at the UKOUG Business Analytics SIG Meeting in London, April 5th 2016, and following on from the recent… Author: Mark Rittman Published: 2016-09-25T01:13:51+01:00 Modified: 2016-09-25T01:13:51+01:00 --- Tomorrow I’m going back to my roots and presenting on Oracle BI at the [UKOUG Business Analytics SIG Meeting](http://www.ukoug.org/events/ukoug-business-analytics-sig-05-04-16/) in London, April 5th 2016, and following on from the recent [Gartner Business Analytics Magic Quadrant 2016](http://blogs.gartner.com/cindi-howson/2016/02/10/biamq/) I’ll be asking — is Oracle BI dead and buried … … or is it more relevant than ever in the world of BiModal IT, and Mode 1 + Mode 2 Analytics? You won’t be surprised to find that I think it’s the latter … as Gartner set out in their [press releases and articles](http://www.gartner.com/newsroom/id/3198917) around Modern BI Platforms and Mode 1/Mode 2 analytics, the key now is for IT departments to **enable** users to perform their own analytics and explore data on their own, not by doing it themselves but by creating a platform, both on-premise and in the cloud, to make this possible. OBIEE12c with it’s strong enterprise background and support for data discovery and external data mashups makes it the ideal platform for delivering Gartner’s vision of a BIModal modern BI platform — and if you’re not at the SIG meeting tomorrow, my slides are on Slideshare here : [https://www.slideshare.net/rittmanmead/oracle-bi-hybrid-bi-mode-1-mode-2-cloud-onpremise-business-analytics](https://www.slideshare.net/rittmanmead/oracle-bi-hybrid-bi-mode-1-mode-2-cloud-onpremise-business-analytics) --- # Presenting Second in the Gluent New World Webinar Series, on SQL-on-Hadoop Concepts and… Canonical: https://rittmananalytics.com/blog/2016/09/25/presenting-second-in-the-gluent-new-world-webinar-series-on-sql-on-hadoop-concepts-and URL: https://rittmananalytics.com/blog/2016/09/25/presenting-second-in-the-gluent-new-world-webinar-series-on-sql-on-hadoop-concepts-and Description: I’m very pleased to be delivering the second in the Gluent New World webinar series, a program of webinars organised by Oracle ACE Director and Gluent CEO Tanel Poder. Tanel… Author: Mark Rittman Published: 2016-09-25T01:07:53+01:00 Modified: 2016-09-25T01:07:53+01:00 --- * * * I’m very pleased to be delivering the second in the Gluent New World webinar series, a program of webinars organised by Oracle ACE Director and Gluent CEO Tanel Poder. Tanel opened the series with a talk about [in-memory processing for databases](http://blog.tanelpoder.com/2016/03/27/gnw01-in-memory-processing-for-databases/), and I’ll be continuing the series with a session on [the rationale and core technical concepts behind my current area of focus — SQL-on-Hadoop](http://blog.tanelpoder.com/2016/04/07/gluent-new-world-02-sql-on-hadoop/). I was initially sceptical when I heard about Apache Hive and vendor-specific implementations but set-based processing, aggregation and querying is a core requirement for virtually every data processing platform — and over the years the core Hive and HCatalog foundation has evolved into technologies such as Spark SQL for large-scale processing, Cloudera Impala and Apache Drill for more interactive workloads, and specialised storage layers and file formats to support the different types of SQL workloads. I’ll also take a look at what new technologies and capabilities are “coming around the corner”, and preview some of the ideas and demos I’ll be showing a the upcoming Enkitec E4 Conference in Barcelona later this month where I’ll be speaking on the same topic. Registration is free and you can sign up here: [https://attendee.gotowebinar.com/register/524034212628017924](https://attendee.gotowebinar.com/register/524034212628017924) Update : The slides from the event are below: --- # From lots of reports (with some data Analysis) to Massive Data Analysis (With some Reporting) Canonical: https://rittmananalytics.com/blog/2016/09/25/from-lots-of-reports-with-some-data-analysis-to-massive-data-analysis-with-some-reporting URL: https://rittmananalytics.com/blog/2016/09/25/from-lots-of-reports-with-some-data-analysis-to-massive-data-analysis-with-some-reporting Description: Slides (and some embedded videos) from my session at the inaugural Danish BI Meetup in Copenhagen, “From lots of reports (with some data Analysis) to Massive Data Analysis (With some… Author: Mark Rittman Published: 2016-09-25T01:02:31+01:00 Modified: 2016-09-25T01:02:31+01:00 --- Slides (and some embedded videos) from my session at the inaugural Danish BI Meetup in Copenhagen, “From lots of reports (with some data Analysis) to Massive Data Analysis (With some Reporting)” — on the future of BI, how Hadoop is the new analytic platform, and how the BI industry is changing with the advent of self-service data discovery tools. [https://www.slideshare.net/rittmanmead/from-lots-of-reports-with-some-data-analysis-to-massive-data-analysis-with-some-reporting](https://www.slideshare.net/rittmanmead/from-lots-of-reports-with-some-data-analysis-to-massive-data-analysis-with-some-reporting) --- # Building Predictive Analytics Models against Wearables, Smart Home and Smartphone App Data — Here’s… Canonical: https://rittmananalytics.com/blog/2016/09/24/building-predictive-analytics-models-against-wearables-smart-home-and-smartphone-app-data-heres URL: https://rittmananalytics.com/blog/2016/09/24/building-predictive-analytics-models-against-wearables-smart-home-and-smartphone-app-data-heres Description: Earlier today I was the guest of Christian Antognini at the Trivadis Tech Event 2016 in Zurich, Switzerland, presenting on the use of Oracle Big Data Discovery as the “data… Author: Mark Rittman Published: 2016-09-25T00:58:50+01:00 Modified: 2016-09-25T00:58:50+01:00 --- ![Building Predictive Analytics Models against Wearables, Smart Home and Smartphone App Data — Here’s…](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2016%2F09%2F1752277806734-no17w.png%3Ffit%3D800%252C394%26ssl%3D1) * * * Earlier today I was the guest of Christian Antognini at the [Trivadis Tech Event 2016](http://m.trivadis.com/techevent) in Zurich, Switzerland, presenting on the use of Oracle Big Data Discovery as the “data scientists’ toolkit”, using data from my fitness devices, smart home appliances and social media activity as the dataset. The slides are here: [https://www.slideshare.net/rittmanmead/using-oracle-big-data-discovey-as-a-data-scientists-toolkit](https://www.slideshare.net/rittmanmead/using-oracle-big-data-discovey-as-a-data-scientists-toolkit) Whilst the main focus of the talk was on the use of BDD for data tidying, data visualisation and machine learning, another part of the talk that people found interesting but that I couldn’t spend much time on was how I got the data into the Hadoop data lake I put together to hold the data. Whilst the main experiment I focused on in the talk just used a simple data export from the Jawbone UP website ([https://jawbone.com](https://jawbone.com/)) to get my workout, sleep and weight data, I’ve since gone on to bring a much wider set of personal data into the data lake via two main routes; ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fblog-images%2F1752277806734-no17w.png%3Fssl%3D1) - IFTTT (“If this, then that”) to capture activity from UP by Jawbone, Strava, Instagram, Twitter and other smartphone apps which it then sends in the form of JSON documents via HTTP GET web requests to a Logstash ingestion server running back home for eventual loading in HDFS, then Hive via Cloudera’s JSONSerde ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fblog-images%2F1752277807077-2huac5.png%3Fssl%3D1) - A Samsung Smart Things “smartapp” that subscribes to smart device and sensor activity in my home, then sends those events as JSON documents again via HTTP GET web requests to Logstash, again for eventual storage into HDFS and Hive and hourly uploads into Big Data Discovery’s DGraph NoSQL engine This all works well and means that my dashboards and analysis in Big Data Discovery are at most one-hour behind the actual activities that take place in my smartphone apps or in my home amongst the various connected heating thermostats, presence sensors and other devices eventually feeding their activity through my Samsung Smart Things hub and the smartapp that I’ve got running in Samsung’s Smart Things cloud service. More recently though I’ve got the same activity also loading into [Apache Kudu](http://kudu.apache.org/), the new real-time fast analytics storage layer for Hadoop, using [a new Hadoop ETL tool called “Streamsets”](http://streamsets.com/) that streams data in real-time into [Kudu and Cloudera Impala](http://blog.cloudera.com/blog/2016/04/how-to-use-impala-and-kudu-together-for-analytic-workloads/) from a variety of source types. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fblog-images%2F1752277807326-6fkvy7.png%3Fssl%3D1) This additional data loading and storage route now gives me metrics in actual real-time (not the one-hour delay caused by BDD’s DGraph hourly batch load), and gives me the total and complete set of events and not the representative sample that Big Data Discovery normally brings in via a DGraph data processing CLI load. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fnfqvbiqopvwmhnrrabjr.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fblog-images%2Fblog-images%2F1752277807586-1xqae.png%3Fssl%3D1) Eventually the aim is to connect it all up via Kafka, but for now it’s a lovely sunny day in Zurich, I’m meeting Christian Berg for a beer and lunch in 30 minutes, so I think it’s time for a well-deserved afternoon-off, and then prep for my final speaking commitment before Oracle Openworld next week … [Dublin on Monday](http://www.meetup.com/hadoop-user-group-ireland/events/232702102/) to talk about Oracle Big Data. --- # Podcast Episode 1 Now Live — Drill to Detail with Mark Rittman, with Special Guest Stewart Bryson Canonical: https://rittmananalytics.com/blog/2016/09/24/podcast-episode-1-now-live-drill-to-detail-with-mark-rittman-with-special-guest-stewart-bryson URL: https://rittmananalytics.com/blog/2016/09/24/podcast-episode-1-now-live-drill-to-detail-with-mark-rittman-with-special-guest-stewart-bryson Description: Whilst I’m working out what my next adventure will be after leaving my role as CTO at Rittman Mead, the Oracle BI, DW + Data Integration consulting company I co-founded… Author: Mark Rittman Published: 2016-09-25T00:49:53+01:00 Modified: 2016-09-25T00:49:53+01:00 --- * * * Whilst I’m working out what my next adventure will be after leaving my role as CTO at Rittman Mead, the Oracle BI, DW + Data Integration consulting company I co-founded back in 2007, I’m pleased to launch the first episode of my new podcast series, [“Drill to Detail”](http://www.drilltodetail.com/), available to [subscribe and download](http://itunes.apple.com/gb/podcast/drill-to-detail-podcast/id1141126960?mt=2) from Apple iTunes. Inspired by the Apple-focused podcast [“The Talk Show” by John Gruber](http://daringfireball.net/thetalkshow/) and with a similar informal but candid style, each episode I’m joined by one of the “movers and shakers” in the BI, DW and Big Data industry to talk about three topics relevant to that particular guest. In the inaugural episode I’m joined by none-other than my old friend and colleague [Stewart Bryson](http://www.linkedin.com/in/stewartbryson), ex-Rittman Mead and currently CTO of his own company Red Pill Analytics, where we talk about the recent Gartner BI & Analytics Magic Quadrant 2016 and what it means for vendors such as Oracle; what the rise of Bi-Modal IT and Mode 2 analytics means for agile BI methodologies; and we return to the Oracle BI, DW and Big Data Reference Architecture both of us contributed to back in 2014 and ask ourselves what worked, what’s changed and how relevant it is today? [Show notes for Episode 1](http://www.drilltodetail.com/podcast/2016/9/11/drill-to-detail-ep1-after-the-gartner-bia-magic-quadrant-2016-with-special-guest-stewart-bryson) are on the Drill to Detail accompanying website, and check back over the coming weeks for further episodes featuring the likes of Dan McClary (ex Oracle Big Data SQL PM, now at Google), Mike Percy (Cloudera Software Engineer working on Apache Kudu, a new distributed storage technology for analytic workloads sponsored by Cloudera), Kent Graziano (Snowflake DB), Paul Sonderegger (ex-Endeca CTO, now Oracle’s Big Data Evangelist) and Cameron Lackpour (Oracle ACE Director and world expert on Essbase). Subscribing is free through the iTunes Podcast Directory, and episodes will run once a week for a while, dropping back to once every two weeks as I get through the episodes I recorded over the summer. --- # Last-Stop Budapest … And Five New BI and Analytics Technologies Coming Soon for Hadoop Canonical: https://rittmananalytics.com/blog/2016/09/25/2016-09-25-2016-09-25-last-stop-budapest-and-five-new-bi-and-analytics-technologies-coming-soon-for-hadoop URL: https://rittmananalytics.com/blog/2016/09/25/2016-09-25-2016-09-25-last-stop-budapest-and-five-new-bi-and-analytics-technologies-coming-soon-for-hadoop Description: Over the past few months I’ve been speaking at a number of user group events in Europe and the Middle East on the future of BI, analytics and data integration… Author: Mark Rittman Published: 2016-09-25T00:05:52+00:00 Modified: 2016-09-25T00:05:52+00:00 --- ![Last-Stop Budapest … And Five New BI and Analytics Technologies Coming Soon for Hadoop](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2F1aa78-69fac78f-4d98-48f2-9c79-27f3beb837e5_779x232.png%3Fssl%3D1) Over the past few months I’ve been speaking at a number of user group events in Europe and the Middle East on the future of BI, analytics and data integration on the Hadoop and NoSQL platform. The original idea behind the presentation was to show people how BI on Hadoop isn’t just slow Apache Hive queries, offloaded ETL work or doing the same old things but with less mature tools. I’ve updated the slides after each presentation and delivered it for the final time today in Budapest, at the [Budapest Data Forum](http://budapestdata.hu/2016/en/) organised by my good friend [Bence Arato](https://www.linkedin.com/in/bencearato). The [final version of the slides](http://www.slideshare.net/rittmanmead/sqlonhadoop-for-analytics-bi-what-are-my-options-whats-the-future) are on Slideshare and you can view and download them using the widget below (and bear in mind I wrote most of the slides back in February 2016, when using Donald Trump to introduce Apache Spark made the audience laugh, not look at each other worried as it did today) [https://www.slideshare.net/rittmanmead/sqlonhadoop-for-analytics-bi-what-are-my-options-whats-the-future](https://www.slideshare.net/rittmanmead/sqlonhadoop-for-analytics-bi-what-are-my-options-whats-the-future) By the time I delivered this final iteration of the session there were so many “next-gen” Hadoop-on-SQL and BI-related technologies in the talk — Tez, Kudu, Spark, Apache Arrow, Apache Lens, Impala, RecordService to pick just a few — I thought it would be worth calling out what I think are the “Top 5” upcoming BI and SQL-related technologies and projects in my talk, starting with my #5… ### 5\. Cloudera Impala, and Parquet Storage — Fast BI Queries for Hadoop Most people using Hadoop for analytic workloads will be aware of [Impala](http://impala.io/), but  it’s importance can’t be understated as the easiest way to improve the response times of BI-type queries when a customer is currently using just Hive. Using a daemon-style architecture similar to the MPP/shared-nothing analytic RDBMSs, moving to Impala from Hive for analytic workloads is easy and most BI tool vendors support Impala as a data source… ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2F1aa78-69fac78f-4d98-48f2-9c79-27f3beb837e5_779x232.png%3Fssl%3D1) …and coupled with the column-oriented [Parquet file storage format](https://parquet.apache.org/), you’re getting close-to or better-than query performance than your old legacy data warehouse platform. ### 4\. Apache Lens — BI Middleware for ROLAP + Business Metadata When you move from a desktop BI tool such as Tableau or Qlikview to an enterprise BI platform such as Oracle BI Enterprise Edition or SAP Business Objects, one of the main benefits to users is the shared business metadata model they provide that abstracts away the various different data sources into more understandable conformed dimensional model. Some provide additional middleware engines that turn this into a fully-fledged ROLAP (Relational OLAP) cube with a dimensionally-orientated query language that the middleware layer then translates into the specific SQL and MDX languages used by the data sources that map into it. [Apache Lens](https://lens.apache.org/), just out of incubator status and now a top-level Apache project, aims to do this for Hadoop. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2F5dd13-39796ca4-a2c5-45c7-992f-37b75ee67cd3_720x540.png%3Fssl%3D1) And again paralleling the product architectures of Oracle Business Intelligence, IBM Cognos and SAP Business Objects, Hadoop now has its own MOLAP (Multidimensional OLAP) server — [Apache Kylin](http://kylin.apache.org/) — and one of the now en-vogue “notebook”-style query tools, [Apache Zeppelin](https://zeppelin.apache.org/), making the next-generation Hadoop BI product architecture look something like the diagram below. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2Fbaee5-3090cd72-6957-41e3-b155-43bc2a0ca7bc_800x599.png%3Fssl%3D1) ### 3\. Apache Arrow — Unified In-Memory Storage Format for Hadoop Whereas Hadoop used to be synonymous with MapReduce and writing everything to disk, Hadoop 2.0 through YARN and in-memory processing frameworks such as [Apache Spark](http://spark.apache.org/) is now all about in-memory — and joins languages and toolkits such as R and Python’s Pandas in doing much of their work in-memory … but things often grind to a halt when you try to move data from one in-memory technology to another as Hadoop serialises and then deserialises the data to move it from one in-memory storage format to another. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2Feeab7-ece31986-66fc-4a85-acae-b7a9b3b431cc_800x437.png%3Fssl%3D1) [Apache Arrow](https://arrow.apache.org/), more of a standard and set of back-end APIs than an end-user BI tool, aims to standardise these in-memory storage formats and take away this data exchange overhead — leading to far greater interoperability between Hadoop’s in-memory tools and languages. ### 2\. Cloudera Kudu — Fast, Real-Time Storage Engine for BI on Hadoop Up until now, if you needed to update, delete or even just insert single rows of data into an Apache Hive table, your best bet was to switch-out the table’s storage format from HDFS to the HBase, a type of NoSQL database that supports random access to individual storage cells and when combined with Hive using the HBase Storage Handler, gives developers a roundabout-way to perform these actions in a Hadoop environment — useful when you need to continue to load SCD2-type dimension tables as part of a data warehouse offload project. But HBase isn’t designed for performing the sort of large-scale aggregations and data selections common in data warehouse projects … and Parquet Storage, which is optimised for these types of queries, doesn’t handle trickle-feed data updates because of the columnar format it uses and the way it compresses data to better optimise it for queries. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2F621a9-49acf503-408c-491b-9f56-e48b21f80b36_731x692.png%3Fssl%3D1) [Apache Kudu](http://getkudu.io/), coming out of Cloudera but now an incubating Apache project, aims to combine HBase’s insert, update and delete capabilities with the fast columnar storage format of Parquet. [Combined with Impala](https://blog.cloudera.com/blog/2016/04/how-to-use-impala-and-kudu-together-for-analytic-workloads/) and with support for other query tools and processing frameworks coming, Kudu is the most ambitious attempt to go beyond simple file-based storage in Hadoop and replace it with any optimised storage layer designed specifically for analytic workloads — and all of it still open-source and free to deploy on as many servers as you need — in contrast to the $1m or so that traditional RDBMS vendors charge for their high-end database servers. ### 1\. Apache Drill — SQL over Self-Describing Data Whilst Hive, and Impala, and all the other SQL-on-Hadoop engines that build on the metastore data dictionary framework that Hive provides are all great ways to run SQL processing on the Hadoop platform, you’re still essentially following the same workflow as you did when working on traditional RDBMS platforms — you need to formally define your table structures before you run your SQL queries, a complicated and often manual process that adds significant time to the data provisioning process for flexible-schema “data lakes” … even though many of the new storage formats we use in Hadoop environments such as Parquet, JSON documents or even CSV files have metadata stored within them. [Apache Drill](https://drill.apache.org/), closely associated with MapR but also now an Apache project, lets you query these “self-describing” data sources directly without requiring you to formally set up their metadata in the Hive metastore … and combine the results with regular Hive tables, data from HBase or even data from traditional database engines such as Oracle, Teradata or Microsoft SQL Server. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2Fdae1b-2e46cd58-8a93-48e8-a48c-06e773f8e4c6_800x372.jpeg%3Fssl%3D1) But it gets better … Drill has the ad-hoc query performance of Impala, Presto and other SQL engines optimised for BI-style workloads, making it an interesting future companion to Apache Hive and supporting data-discovery style applications up until now only possible through proprietary vendor tools like Oracle’s Endeca Information Discovery ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2F9f885-efe356d2-0a81-4d50-8cb3-606ae4d5c9a9_800x222.jpeg%3Fssl%3D1) Of course there’s many more new Hadoop projects coming down-the-line that are concerned with BI, along with hot new Hadoop startups such as [Gluent](http://gluent.com/) that aim to bridge the gap between “old world” RDBMS and “new world” Hadoop and NoSQL storage and compute frameworks — a company and development team I suspect we’ll hear a lot more about in the future. For now though, those are the top 5 current and future BI technologies I’ve been talking about on my travels over the past six months … now on to something even more interesting. --- # Building Predictive Analytics Models against Wearables, Smart Home and Smartphone App Data — Here’s… Canonical: https://rittmananalytics.com/blog/2016/09/24/2016-09-24-2016-09-24-building-predictive-analytics-models-against-wearables-smart-home-and-smartphone-app-data-heres URL: https://rittmananalytics.com/blog/2016/09/24/2016-09-24-2016-09-24-building-predictive-analytics-models-against-wearables-smart-home-and-smartphone-app-data-heres Description: Earlier today I was the guest of Christian Antognini at the Trivadis Tech Event 2016 in Zurich, Switzerland, presenting on the use of Oracle Big Data Discovery as the “data… Author: Mark Rittman Published: 2016-09-24T23:58:50+00:00 Modified: 2016-09-24T23:58:50+00:00 --- ![Building Predictive Analytics Models against Wearables, Smart Home and Smartphone App Data — Here’s…](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2F3cc80-57247264-3e94-4e8e-b926-2c530603eb21_800x394.png%3Fssl%3D1) * * * Earlier today I was the guest of Christian Antognini at the [Trivadis Tech Event 2016](http://m.trivadis.com/techevent) in Zurich, Switzerland, presenting on the use of Oracle Big Data Discovery as the “data scientists’ toolkit”, using data from my fitness devices, smart home appliances and social media activity as the dataset. The slides are here: [https://www.slideshare.net/rittmanmead/using-oracle-big-data-discovey-as-a-data-scientists-toolkit](https://www.slideshare.net/rittmanmead/using-oracle-big-data-discovey-as-a-data-scientists-toolkit) Whilst the main focus of the talk was on the use of BDD for data tidying, data visualisation and machine learning, another part of the talk that people found interesting but that I couldn’t spend much time on was how I got the data into the Hadoop data lake I put together to hold the data. Whilst the main experiment I focused on in the talk just used a simple data export from the Jawbone UP website ([https://jawbone.com](https://jawbone.com/)) to get my workout, sleep and weight data, I’ve since gone on to bring a much wider set of personal data into the data lake via two main routes; ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2F3cc80-57247264-3e94-4e8e-b926-2c530603eb21_800x394.png%3Fssl%3D1) - IFTTT (“If this, then that”) to capture activity from UP by Jawbone, Strava, Instagram, Twitter and other smartphone apps which it then sends in the form of JSON documents via HTTP GET web requests to a Logstash ingestion server running back home for eventual loading in HDFS, then Hive via Cloudera’s JSONSerde ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2Fac3e0-02992164-73f8-4ee7-97b3-a76a03348562_800x518.png%3Fssl%3D1) - A Samsung Smart Things “smartapp” that subscribes to smart device and sensor activity in my home, then sends those events as JSON documents again via HTTP GET web requests to Logstash, again for eventual storage into HDFS and Hive and hourly uploads into Big Data Discovery’s DGraph NoSQL engine This all works well and means that my dashboards and analysis in Big Data Discovery are at most one-hour behind the actual activities that take place in my smartphone apps or in my home amongst the various connected heating thermostats, presence sensors and other devices eventually feeding their activity through my Samsung Smart Things hub and the smartapp that I’ve got running in Samsung’s Smart Things cloud service. More recently though I’ve got the same activity also loading into [Apache Kudu](http://kudu.apache.org/), the new real-time fast analytics storage layer for Hadoop, using [a new Hadoop ETL tool called “Streamsets”](http://streamsets.com/) that streams data in real-time into [Kudu and Cloudera Impala](http://blog.cloudera.com/blog/2016/04/how-to-use-impala-and-kudu-together-for-analytic-workloads/) from a variety of source types. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2Fb328e-f2e45140-1d93-47ee-9ae7-d3649f9340a7_800x448.png%3Fssl%3D1) This additional data loading and storage route now gives me metrics in actual real-time (not the one-hour delay caused by BDD’s DGraph hourly batch load), and gives me the total and complete set of events and not the representative sample that Big Data Discovery normally brings in via a DGraph data processing CLI load. ![](https://rittmananalytics.com/api/public/og-image?src=https%3A%2F%2Fi0.wp.com%2Fdtdtemp8.wpcomstaging.com%2Fwp-content%2Fuploads%2F2025%2F06%2F99310-a8505bc5-dc1e-4811-ada9-7f6b1de0162a_800x256.png%3Fssl%3D1) Eventually the aim is to connect it all up via Kafka, but for now it’s a lovely sunny day in Zurich, I’m meeting Christian Berg for a beer and lunch in 30 minutes, so I think it’s time for a well-deserved afternoon-off, and then prep for my final speaking commitment before Oracle Openworld next week … [Dublin on Monday](http://www.meetup.com/hadoop-user-group-ireland/events/232702102/) to talk about Oracle Big Data. --- # Podcast Episode 1 Now Live — Drill to Detail with Mark Rittman, with Special Guest Stewart Bryson Canonical: https://rittmananalytics.com/blog/2016/09/24/2016-09-24-2016-09-24-podcast-episode-1-now-live-drill-to-detail-with-mark-rittman-with-special-guest-stewart-bryson URL: https://rittmananalytics.com/blog/2016/09/24/2016-09-24-2016-09-24-podcast-episode-1-now-live-drill-to-detail-with-mark-rittman-with-special-guest-stewart-bryson Description: Whilst I’m working out what my next adventure will be after leaving my role as CTO at Rittman Mead, the Oracle BI, DW + Data Integration consulting company I co-founded… Author: Mark Rittman Published: 2016-09-24T23:49:53+00:00 Modified: 2016-09-24T23:49:53+00:00 --- * * * Whilst I’m working out what my next adventure will be after leaving my role as CTO at Rittman Mead, the Oracle BI, DW + Data Integration consulting company I co-founded back in 2007, I’m pleased to launch the first episode of my new podcast series, [“Drill to Detail”](http://www.drilltodetail.com/), available to [subscribe and download](http://itunes.apple.com/gb/podcast/drill-to-detail-podcast/id1141126960?mt=2) from Apple iTunes. Inspired by the Apple-focused podcast [“The Talk Show” by John Gruber](http://daringfireball.net/thetalkshow/) and with a similar informal but candid style, each episode I’m joined by one of the “movers and shakers” in the BI, DW and Big Data industry to talk about three topics relevant to that particular guest. In the inaugural episode I’m joined by none-other than my old friend and colleague [Stewart Bryson](http://www.linkedin.com/in/stewartbryson), ex-Rittman Mead and currently CTO of his own company Red Pill Analytics, where we talk about the recent Gartner BI & Analytics Magic Quadrant 2016 and what it means for vendors such as Oracle; what the rise of Bi-Modal IT and Mode 2 analytics means for agile BI methodologies; and we return to the Oracle BI, DW and Big Data Reference Architecture both of us contributed to back in 2014 and ask ourselves what worked, what’s changed and how relevant it is today? [Show notes for Episode 1](http://www.drilltodetail.com/podcast/2016/9/11/drill-to-detail-ep1-after-the-gartner-bia-magic-quadrant-2016-with-special-guest-stewart-bryson) are on the Drill to Detail accompanying website, and check back over the coming weeks for further episodes featuring the likes of Dan McClary (ex Oracle Big Data SQL PM, now at Google), Mike Percy (Cloudera Software Engineer working on Apache Kudu, a new distributed storage technology for analytic workloads sponsored by Cloudera), Kent Graziano (Snowflake DB), Paul Sonderegger (ex-Endeca CTO, now Oracle’s Big Data Evangelist) and Cameron Lackpour (Oracle ACE Director and world expert on Essbase). Subscribing is free through the iTunes Podcast Directory, and episodes will run once a week for a while, dropping back to once every two weeks as I get through the episodes I recorded over the summer. --- # Drill to Detail Ep.121 'How Partnerships Power the Data Analytics Ecosystem' featuring Special Guest Eleanor Thompson Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A67d1d4b415d4e45da4bba940 URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A67d1d4b415d4e45da4bba940 Description: Mark Rittman is joined on this episode by Branchworks CEO Eleanor Thompson to talk about partner programs, partner portals and the role partnerships play in making data analytics initiatives a success. Branchworks homepage : Activating Partnerships for Revenue Growth CastorDoc case study Eleanor Tho Show: Drill to Detail Published: 2025-03-12T18:38:44+00:00 --- Mark Rittman is joined on this episode by Branchworks CEO Eleanor Thompson to talk about partner programs, partner portals and the role partnerships play in making data analytics initiatives a success. [Branchworks homepage : Activating Partnerships for Revenue Growth](https://branchworks.io/) [CastorDoc case study](https://branchworks.io/case-studies/castordoc/) [Eleanor Thompson LinkedIn Profile](https://www.linkedin.com/in/eleanorthompson10/) --- # Drill to Detail Ep.120 ‘Embeddable, Cube and Lightning-Fast In-App Analytics’ with Special Guest Tom Gardiner Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A67bfa32e44df6214bef6faec URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A67bfa32e44df6214bef6faec Description: In this episode of the Drill to Detail Podcast Mark Rittman is joined by special guest Tom Gardiner, CEO @ Embeddable.com to talk about Embeddable's mission to create the leading developer toolkit for building fast, interactive customer-facing analytics directly into your product in partnership with Show: Drill to Detail Published: 2025-02-26T23:26:38+00:00 --- In this episode of the Drill to Detail Podcast Mark Rittman is joined by special guest Tom Gardiner, CEO @ Embeddable.com to talk about Embeddable's mission to create the leading developer toolkit for building fast, interactive customer-facing analytics directly into your product in partnership with Cube.dev and Rittman Analytics. [Trevor.io](https://trevor.io/) [Embeddable.com](https://embeddable.com/) [Why Embeddable?](https://embeddable.com/why) --- # Drill to Detail Ep.119 ‘Data Control Planes, SDF Labs and dbt Cloud Visual Editor’ with Special Guest Greg McKeon Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A67ad1a1ec487797349b80eaf URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A67ad1a1ec487797349b80eaf Description: Mark Rittman is joined in this episode by Greg McKeon, Staff Product Manager at dbt Labs to talk about their recent acquisition of SDF Labs, the vision for dbt as the control plane for data collaboration at-scale and the upcoming drag-and-drop Visual Editor that’s soon to be part of dbt Cloud. dbt C Show: Drill to Detail Published: 2025-02-12T22:01:02+00:00 --- Mark Rittman is joined in this episode by Greg McKeon, Staff Product Manager at dbt Labs to talk about their recent acquisition of SDF Labs, the vision for dbt as the control plane for data collaboration at-scale and the upcoming drag-and-drop Visual Editor that’s soon to be part of dbt Cloud. [dbt Cloud: The control plane for data collaboration at scale](https://www.getdbt.com/resources/whitepaper-the-control-plane-for-data-collaboration-at-scale) [About the Visual Editor](https://docs.getdbt.com/docs/cloud/visual-editor) [Coalesce 2024 and the Launch of dbt](https://rittmananalytics.com/blog/2024/10/11/coalesce-2024-and-the-launch-of-dbts-visual-editing-experience) --- # Drill to Detail Ep.118 ‘A Look Into the Future of Looker and Google Cloud Data Analytics’ featuring Special Guest Sean Zinsmeister Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A67a139f4a2da1b74d731ce44 URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A67a139f4a2da1b74d731ce44 Description: In this episode, Sean Zinsmeister from Google joins Mark Rittman to discuss the latest developments for Looker including the integration of Looker Studio, new modeling capabilities and the exciting potential of generative AI for BI. We discuss how Looker is evolving to be a more open, composable pla Show: Drill to Detail Published: 2025-02-03T21:49:40+00:00 --- In this episode, Sean Zinsmeister from Google joins Mark Rittman to discuss the latest developments for Looker including the integration of Looker Studio, new modeling capabilities and the exciting potential of generative AI for BI. We discuss how Looker is evolving to be a more open, composable platform that can power advanced analytics and data storytelling, with Sean sharing insights on Google's purpose-built Gemini models for natural language to SQL translation and how Looker customers can leverage these AI capabilities.  We also explore Looker's agentic API strategy, the long-term vision of using Looker Studio as the primary Looker front-end and the opening up of LookML to tools beyond just Looker. --- # Drill to Detail Ep.117 ‘How DataCoves Operationalises the Modern Data Stack’ featuring Special Guest Noel Gomez Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A6765b5433d52b61c2f566670 URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A6765b5433d52b61c2f566670 Description: Join Mark Rittman in this special end-of-year episode as he speaks with Noel Gomez, co-founder of DataCoves about the challenges and opportunities of orchestrating dbt and other tools within the open-source Modern Data Stack, navigating the evolving semantic layer landscape and the future of modular Show: Drill to Detail Published: 2024-12-20T18:36:06+00:00 --- Join Mark Rittman in this special end-of-year episode as he speaks with Noel Gomez, co-founder of DataCoves about the challenges and opportunities of orchestrating dbt and other tools within the open-source Modern Data Stack, navigating the evolving semantic layer landscape and the future of modular, vendor-agnostic data solutions. [Datacoves Platform Overview](https://datacoves.com/) [Build vs Buy Analytics Platform: Hosting Open-Source Tools](https://datacoves.com/post/build-vs-buy-analytics) [Scale the benefits of Core with dbt Cloud](https://www.getdbt.com/product/dbt-core-vs-dbt-cloud) --- # Drill to Detail Ep.116 ‘Spotify, Semantic Layers and Steep’s Metrics-First Approach to BI Tools’ with Special Guest Johan Baltzar Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A673bd2737ffc41649e8df258 URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A673bd2737ffc41649e8df258 Description: Mark is joined in this episode by Johan Baltzar, previously Product Analytics Manager at Spotify and now co-founder and CEO at Steep to talk about the role analytics played in Spotify’s growth story, the startup scene in Stockholm, Sweden and Steep’s metrics-first approach to user-centric business a Show: Drill to Detail Published: 2024-11-18T23:57:31+00:00 --- Mark is joined in this episode by Johan Baltzar, previously Product Analytics Manager at Spotify and now co-founder and CEO at Steep to talk about the role analytics played in Spotify’s growth story, the startup scene in Stockholm, Sweden and Steep’s metrics-first approach to user-centric business analytics. [The Kry founders factory: Meet 15 employees-turned-founders](https://sifted.eu/articles/kry-founders-factory) [Steep](https://steep.app/) homepage New in Steep : [Cube](https://steep.app/blog/january-2024) and [dbt](https://steep.app/blog/dbt-cloud-integration%20integration)   [Targets & BI - How hard can it be?](https://steep.app/blog/targets-and-bi) --- # Drill to Detail Ep.115 ‘Airbnb, DataOps and SQLMesh’s Data Engineering Innovation’ with Special Guest Toby Mao Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A67241e292e20f71f1e1755f5 URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A67241e292e20f71f1e1755f5 Description: Mark is joined in this latest Drill to Detail episode by Tobias (Toby) Mao, CTO and co-Founder at Tobiko Data to talk about SQLGlot, the innovation culture at Airbnb and the data engineering challenges solved by SQLMesh. SQLGlot Introducing Minerva — Airbnb’s Metric Platform SQLMesh homepage Tobiko Show: Drill to Detail Published: 2024-11-01T00:17:45+00:00 --- Mark is joined in this latest Drill to Detail episode by Tobias (Toby) Mao, CTO and co-Founder at Tobiko Data to talk about SQLGlot, the innovation culture at Airbnb and the data engineering challenges solved by SQLMesh. [SQLGlot](https://github.com/tobymao/sqlglot) [Introducing Minerva — Airbnb’s Metric Platform](https://medium.com/airbnb-engineering/how-airbnb-achieved-metric-consistency-at-scale-f23cc53dea70) [SQLMesh homepage](https://sqlmesh.com/) [Tobiko Cloud](https://tobikodata.com/product.html) [Running dbt in SQLMesh](https://sqlmesh.readthedocs.io/en/stable/integrations/dbt/) [dbt Mesh](https://www.getdbt.com/product/dbt-mesh) [Dlt (Data Load Tool)](https://dlthub.com/) --- # Drill to Detail Ep.114 ‘Building a Solopreneur Data Analytics Consultancy’ with Special Guest Christian Steinert Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A671954609670760d6d0b0d56 URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A671954609670760d6d0b0d56 Description: Mark is joined in this episode by Christian Steinert to talk about his solo journey building Steinert Analytics, a data analytics consultancy transforming data into actionable revenue and cost-saving insights for Central Ohio's SaaS startups. Steinert Analytics homepage Christian Steinert LinkedIn P Show: Drill to Detail Published: 2024-10-23T20:01:59+00:00 --- Mark is joined in this episode by Christian Steinert to talk about his solo journey building Steinert Analytics, a data analytics consultancy transforming data into actionable revenue and cost-saving insights for Central Ohio's SaaS startups. [Steinert Analytics](https://www.steinertanalytics.com/) homepage [Christian Steinert](https://www.linkedin.com/in/christiansteinert96/) LinkedIn Profile [“Helping a Roofing Company Define Their Sales Process”](https://www.theyrecounting.com/p/solving-business-problems-data-christian-steinert) [“Transforming a Top 5 Fast Food Company's Operational Drive-Thru Reporting”](https://www.steinertanalytics.com/_files/ugd/91ae65_e03d71169e664eb9ab56b19d648ed41b.pdf) --- # Drill to Detail Ep. 113 ‘Data Teams, Portable and Data Integration’s Long Tail’ featuring Special Guest Ethan Aaron Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A66fc40421b06b274a852157e URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A66fc40421b06b274a852157e Description: Mark Rittman is joined in this episode by Ethan Aaron, Founder & CEO of Portable, to talk about what you should do if you’re the first data hire at a company, when and when not to hire a consultant, the founding story of Portable.io and the long tail (and economic model) of the data integration conn Show: Drill to Detail Published: 2024-10-02T18:32:00+00:00 --- Mark Rittman is joined in this episode by Ethan Aaron, Founder & CEO of Portable, to talk about what you should do if you’re the first data hire at a company, when and when not to hire a consultant, the founding story of Portable.io and the long tail (and economic model) of the data integration connectors market. Ethan Aaron [LinkedIn Profile](https://www.linkedin.com/in/ethanaaron/) [Portable.io](https://portable.io/) Homepage [“The biggest misconceptions about data integrations..”](https://www.linkedin.com/posts/ethanaaron_the-biggest-misconceptions-about-data-integrations-activity-7240076657888305152-A3pp?utm_source=share&utm_medium=member_desktop) (LinkedIn) [“You join a 100 person](https://www.linkedin.com/posts/ethanaaron_you-join-a-100-person-company-as-the-head-activity-7225138777030041604-9kBS?utm_source=share&utm_medium=member_desktop) --- # Drill to Detail Ep.112 ‘From Delphi to Cube’s New Semantic Model AI Features’ with Special Guest David Jayatillake Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A66e34e3fd7cfdc727def137b URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A66e34e3fd7cfdc727def137b Description: Mark Rittman is joined by returning guest David Jayatillake, VP of AI at Cube.dev, to talk about Delphi Labs’ journey from a standalone data analytics chatbot to now becoming the basis of Cube’s new AI features within its composable semantic model product. Drill to Detail Ep.102 'LLMs, Semantic Mode Show: Drill to Detail Published: 2024-09-12T20:25:35+00:00 --- Mark Rittman is joined by returning guest David Jayatillake, VP of AI at Cube.dev, to talk about Delphi Labs’ journey from a standalone data analytics chatbot to now becoming the basis of Cube’s new AI features within its composable semantic model product. [Drill to Detail Ep.102 'LLMs, Semantic Models and Bringing AI to the Modern Data Stack' with Special Guest David Jayatillake](https://www.rittmananalytics.com/drilltodetail/2023/4/14/ep102-llms-semantic-models-and-bringing-ai-to-the-modern-data-stack-with-special-guest-david-jayatillake) [Drill to Detail Ep.107 'Cube, Headless BI and the AI Semantic Layer' with Special Guest Artyom Keydunov](https://www.rittmananalytics.com/drilltodetail/2023/7/6/drill-to-detail-ep107-cube-headless-bi-and-the-ai-semantic-layer-with-special-guest-artyom-keydunov) [Introduc](https://cube.dev/blog/introducing-the-ai-api-and-chart-prototyping-in-cube-cloud) --- # Drill to Detail Ep.111 'Making Analytics Fun, Frictionless and Ducking Awesome' with Special Guest Jordan Tigani Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A64fa45bece27571f8c872903 URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A64fa45bece27571f8c872903 Description: Mark Rittman is joined in this final episode of the current series of Drill to Detail by returning guest Jordan Tigani, Co-Founder & CEO at Motherduck to talk about the journey from big data to small data and bringing hybrid cloud execution to DuckDB. Big Data is Dead Teach Your Duckdb To Fly Announ Show: Drill to Detail Published: 2023-09-07T21:50:54+00:00 --- Mark Rittman is joined in this final episode of the current series of Drill to Detail by returning guest Jordan Tigani, Co-Founder & CEO at Motherduck to talk about the journey from big data to small data and bringing hybrid cloud execution to DuckDB. [Big Data is Dead](https://www.datacouncil.ai/talks/big-data-is-dead) [Teach Your Duckdb To Fly](https://motherduck.com/) [Announcing Motherduck: Hybrid Execution Scales Duckdb From Your Laptop Into The Cloud](https://motherduck.com/blog/announcing-motherduck-duckdb-in-the-cloud/) [Drill to Detail Ep.64 ‘Goo](https://www.rittmananalytics.com/drilltodetail/2019/4/29/drill-to-detail-ep64-google-bigquery-bi-engine-and-the-future-of-data-warehousing-with-special-guest-jordan-tigani) --- # Drill to Detail Ep.110 'Building a Sequel to SQL with Malloy' featuring Special Guests Lloyd Tabb and Carlin Eng Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A64da994dc9cdc97d1e9b1544 URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A64da994dc9cdc97d1e9b1544 Description: Mark Rittman is joined in this special episode by previous Looker founder and inventor of the LookML language Lloyd Tabb together with Carlin Eng, Product Manager at Google Cloud to talk about Malloy, a new query language to replace SQL for analytics. Drill to Detail Ep.60 'A Deeper Look Into Looker Show: Drill to Detail Published: 2023-08-14T21:14:53+00:00 --- Mark Rittman is joined in this special episode by previous Looker founder and inventor of the LookML language Lloyd Tabb together with Carlin Eng, Product Manager at Google Cloud to talk about Malloy, a new query language to replace SQL for analytics. [Drill to Detail Ep.60 'A Deeper Look Into Looker' With Special Guest Lloyd Tabb](https://www.rittmananalytics.com/drilltodetail/2019/3/4/drill-to-detail-ep60-a-deeper-look-into-looker-with-special-guest-lloyd-tabb) [Malloy : An experimental language for data](https://www.malloydata.dev/) [A Sequel to SQL? An introduction to Malloy](https://carlineng.com/?postid=malloy-intro#blog) [Why SQL syntax sucks, and why it matters](https://carlineng.com/?postid=sql-bad-syntax#blog) --- # Drill to Detail Ep.109 'Dagster, Orchestration and Software-Defined Assets' with Special Guest Nick Schrock Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A64cb5caf10b341420b7c29cc URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A64cb5caf10b341420b7c29cc Description: Mark Rittman is joined in this episode by returning guest and Elementl Founder Nick Shrock to talk about Dagster's role in the modern data stack ecosystem and software defined assets, a new, declarative approach to managing data and orchestrating its maintenance. Introducing Software-Defined Assets Show: Drill to Detail Published: 2023-08-03T07:52:15+00:00 --- Mark Rittman is joined in this episode by returning guest and Elementl Founder Nick Shrock to talk about Dagster's role in the modern data stack ecosystem and software defined assets, a new, declarative approach to managing data and orchestrating its maintenance. [Introducing Software-Defined Assets](https://dagster.io/blog/software-defined-assets) [Rethinking Orchestration as Reconciliation: Software-Defined Assets in Dagster](https://www.youtube.com/watch?v=-OIIbolV6iA) [Optimizing Data Materialization Using Dagster’s Policies](https://blog.rittmananalytics.com/optimizing-data-materialization-using-dagsters-policies-308fc77aba3c) --- # Drill to Detail Ep.108 "Mode, ThoughtSpot and AI as the Next Discontinuity" with Special Guest Benn Stancil Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A64b96a211e962771b5a71a36 URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A64b96a211e962771b5a71a36 Description: Mark Rittman is joined in this special episode of Drill to Detail by Benn Stancil, CTO and Founder at Mode to talk about their recent acquisition by ThoughtSpot, the economics of the modern data stack and how AI and LLMs could likely become our industry’s next discontinuity. We don’t need another SQ Show: Drill to Detail Published: 2023-07-20T17:19:17+00:00 --- Mark Rittman is joined in this special episode of Drill to Detail by Benn Stancil, CTO and Founder at Mode to talk about their recent acquisition by ThoughtSpot, the economics of the modern data stack and how AI and LLMs could likely become our industry’s next discontinuity. [We don’t need another SQL chatbot](https://benn.substack.com/p/we-dont-need-another-sql-chatbot) [The new philosophers](https://benn.substack.com/p/the-new-philosophers) [ThoughtSpot acquires Mode Analytics, a BI platform, for $200M in cash and stock](https://techcrunch.com/2023/06/26/thoughtspot-acquires-mode-analytics-a-bi-platform-for-200m-in-cash-and-stock) --- # Drill to Detail Ep.107 'Cube, Headless BI and the AI Semantic Layer' with Special Guest Artyom Keydunov Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A64a6e32ebfd9cd3432613b64 URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A64a6e32ebfd9cd3432613b64 Description: Mark Rittman is joined in this episode by Artyom Keydunov, Founder at Cube to talk about embedded analytics and Cube's origin story, headless BI, query acceleration and how Cube and Delphi enable an AI-powered conversational interface for the semantic layer. Empower uniquely insightful AI & LLM data Show: Drill to Detail Published: 2023-07-06T15:58:54+00:00 --- Mark Rittman is joined in this episode by Artyom Keydunov, Founder at Cube to talk about embedded analytics and Cube's origin story, headless BI, query acceleration and how Cube and Delphi enable an AI-powered conversational interface for the semantic layer. [Empower uniquely insightful AI & LLM data experiences](https://cube.dev/use-cases/llm-and-ai-semantic-layer) [AI-powered conversational interface for the semantic layer](https://cube.dev/blog/conversational-interface-for-semantic-layer) [Define metrics upstream to align your team](https://cube.dev/use-cases/semantic-layer) [How Rittman Analytics Delivers the Semantic Layer Today with Cube](https://youtu.be/EWZmQlCvX3M) --- # Drill to Detail Ep.106 'Customer Studio, Hightouch Performance and the Evolution of Reverse ETL' with Special Guest Tejas Manohar Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A6494c44fcebe461ef111f5a4 URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A6494c44fcebe461ef111f5a4 Description: Hightouch co-Founder and co-CEO Tejas Manohar returns as special guest to talk with Mark Rittman about the reverse ETL market today, the evolution of the composable customer data platform and new featured in Hightouch to enrich customer profiles and drive personalization across marketing campaigns. Show: Drill to Detail Published: 2023-06-22T22:11:18+00:00 --- Hightouch co-Founder and co-CEO Tejas Manohar returns as special guest to talk with Mark Rittman about the reverse ETL market today, the evolution of the composable customer data platform and new featured in Hightouch to enrich customer profiles and drive personalization across marketing campaigns. [Reverse ETL is Dead](https://www.linkedin.com/posts/ethanaaron_reverse-etl-is-dead-all-the-vendors-have-activity-7065774434946576384-6SsV?utm_medium=member_desktop&utm_source=share) (Ethan Aaron LinkedIn Post) [Customer 360 Data Warehousing and Sync to Hubspot](https://www.rittmananalytics.com/blog/2021/2/15/customer-360-degree-analysis-and-hightouch) [You don't need the Modern Data Stack to get sh\*t done](https://hightouch.com/blog/you-dont-need-the-mds) --- # Drill to Detail Ep.105 ‘Data Catalogs, Data Discovery and Data Lineage for the Modern Data Stack’ with Special Guest Shinji Kim Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A647918948dc23632f3eebdb6 URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A647918948dc23632f3eebdb6 Description: Mark is joined by Shinji Kim, Founder and CEO of Select Star to talk about their mission to re-invent data catalogs, data discovery and data lineage for the modern data stack Data Discovery vs. Data Observability: Understanding the Differences for Better DataOps Select Star and dbt Labs Partner for Show: Drill to Detail Published: 2023-06-01T22:15:47+00:00 --- Mark is joined by Shinji Kim, Founder and CEO of Select Star to talk about their mission to re-invent data catalogs, data discovery and data lineage for the modern data stack [Data Discovery vs. Data Observability: Understanding the Differences for Better DataOps](https://www.selectstar.com/blog/data-discovery-vs-data-observability-understanding-the-differences-for-better-dataops) [Select Star and dbt Labs Partner for Better Data Discovery on dbt](https://www.selectstar.com/blog/select-star-and-dbt-partner-for-column-lineage-and-data-discovery) [Select Star : Free Trial](https://app.selectstar.com/signup?_gl=1%2Asxtpvv%2A_gcl_au%2ANzEwMjEzODI1LjE2ODU2NTcyODE.) --- # Drill to Detail Ep.104 ‘Delta Lakes, Tiger Temples and the Databricks Lakehouse Platform’ with Special Guests Jason Pohl and Stewart Bryson Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A64654fb4c0580c653a549238 URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A64654fb4c0580c653a549238 Description: Mark Rittman is joined by Jason Pohl, Senior Director of Data Management at Databricks along with special co-host and Head of Customer Experience at Coalesce, Stewart Bryson to talk about data lakehouses, Databricks and helping to raise tigers in a Thai Tiger Temple. Databricks - A History What is a Show: Drill to Detail Published: 2023-05-17T22:19:47+00:00 --- Mark Rittman is joined by Jason Pohl, Senior Director of Data Management at Databricks along with special co-host and Head of Customer Experience at Coalesce, Stewart Bryson to talk about data lakehouses, Databricks and helping to raise tigers in a Thai Tiger Temple. [Databricks - A History](https://medium.com/@chuck.connell.3/databricks-a-history-d8dd12fe9695) [What is a Delta Lake?](https://docs.databricks.com/delta/index.html) [The Databricks Lakehouse Platform](https://www.databricks.com/product/data-lakehouse) [Databricks SQL](https://www.databricks.com/product/databricks-sql) [Jason Pohl LinkedIn Profile](https://www.linkedin.com/in/jasonpohl/) --- # Drill to Detail Ep. 103 'Reverse ETL, Profiles Sync and Segment Unify' with Special Guest Kevin Niparko Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A64495b57b9cdf4347e36dab6 URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A64495b57b9cdf4347e36dab6 Description: Mark Rittman is joined by Twilio Segment Head of Product Kevin Niparko to talk about trends in the customer data platform market, Reverse ETL and Profiles Sync, the impact of LLMs (Large Language Models) on digital customer experience and Segment Unify, a consumer scale real-time identity resolution Show: Drill to Detail Published: 2023-04-26T17:11:00+00:00 --- Mark Rittman is joined by Twilio Segment Head of Product Kevin Niparko to talk about trends in the customer data platform market, Reverse ETL and Profiles Sync, the impact of LLMs (Large Language Models) on digital customer experience and Segment Unify, a consumer scale real-time identity resolution solution that provides complete, real-time, portable customer profiles. [Segment Unify is here: complete, real-time, portable customer profiles](https://segment.com/blog/segment-unify-ga/) [Activate warehouse data in all of your destination tools](https://segment.com/product/reverse-etl) [Customer profiles made portable](https://segment.com/product/profiles-sync/) --- # Drill to Detail Ep.102 'LLMs, Semantic Models and Bringing AI to the Modern Data Stack' with Special Guest David Jayatillake Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A64392631a9fe016580028ece URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A64392631a9fe016580028ece Description: Mark Rittman is joined by David Jayatillake, CEO and co-founder of Delphi Labs, to talk about the role of semantic models and the marketplace today, large language models and the phenomena that is ChatGPT, and how Delphi Labs are planning on bringing AI to Slack and the modern data stack. https://ww Show: Drill to Detail Published: 2023-04-14T10:08:49+00:00 --- Mark Rittman is joined by David Jayatillake, CEO and co-founder of Delphi Labs, to talk about the role of semantic models and the marketplace today, large language models and the phenomena that is ChatGPT, and how Delphi Labs are planning on bringing AI to Slack and the modern data stack. [https://www.delphihq.com/#about](https://www.delphihq.com/#about) [https://www.linkedin.com/in/david-jayatillake/details/experience/](https://www.linkedin.com/in/david-jayatillake/details/experience/) [For B2B Generative AI Apps, Is Less More?](https://a16z.com/2023/03/30/b2b-generative-ai-synthai/) --- # Drill to Detail Ep.101 ‘The Story Behind the Story of RJ Metrics, Fishtown Analytics and dbt Labs’ with Special Guest Tristan Handy Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A641b9322d3cc007601c728b4 URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A641b9322d3cc007601c728b4 Description: Joining Mark Rittman for this 101st Episode Special for the Drill to Detail Podcast is Tristan Handy, CEO and Founder of dbt Labs talking about what went right at RJ Metrics, how the Analyst Collective led to today’s community around the open-source dbt project and his personal journey from being in Show: Drill to Detail Published: 2023-03-22T23:45:38+00:00 --- Joining Mark Rittman for this 101st Episode Special for the Drill to Detail Podcast is Tristan Handy, CEO and Founder of dbt Labs talking about what went right at RJ Metrics, how the Analyst Collective led to today’s community around the open-source dbt project and his personal journey from being in the lab building Fishtown Analytics to CEO of today’s hottest data analytics startup … and why he secretly wishes he was Mark (according to Mark). [Ep.100 Special ‘Past, Present and Future of the Modern Data Stack’ with Special Guests Keenan Rice, Stewart Bryson and Jake Stein](https://www.rittmananalytics.com/drilltodetail/2023/3/7/drill-to-detail-ep100-special-past-present-and-future-of-the-modern-data-stack-with-special-guests-keenan-rice-stewart-bryson-and-jake-stein) (Drill to Detail Podcast) --- # Drill to Detail Ep.100 Special ‘Past, Present and Future of the Modern Data Stack’ with Special Guests Keenan Rice, Stewart Bryson and Jake Stein Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A640722fe00c0a0548184baef URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A640722fe00c0a0548184baef Description: Joining Mark Rittman for this 100th Episode Special are Keenan Rice (previously Founding Team, GTM Executive at Looker, now GM at Firebolt), Stewart Bryson (previously CEO at Red Pill Analytics, now Head of Customer Experience at Coalesce) and joining us mid-way through the show as our even more spe Show: Drill to Detail Published: 2023-03-07T11:41:50+00:00 --- Joining Mark Rittman for this 100th Episode Special are Keenan Rice (previously Founding Team, GTM Executive at Looker, now GM at Firebolt), Stewart Bryson (previously CEO at Red Pill Analytics, now Head of Customer Experience at Coalesce) and joining us mid-way through the show as our even more special mystery guest, Jake Stein (previously Co-Founder and COO at RJ Metrics, CEO and Co-Founder at Stitch and now CEO and Co-Founder at Common Paper. Drill to Detail Past Episodes discussed on the show were: [Drill To Detail Ep.1. 'After The Gartner BI&A Magic Quadrant 2016', With Special Guest Stewart Bryson](https://www.rittmananalytics.com/drilltodetail/2016/9/11/drill-to-detail-ep1-after-the-gartner-bia-magic-quadrant-2016-with-special-guest-stewart-bryson) --- # Drill to Detail Ep.99 “Is the Modern Data Stack Dead?” with Special Guest Chris Tabb Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63ed2ace3ef8e80f85a0506d URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63ed2ace3ef8e80f85a0506d Show: Drill to Detail Published: 2023-02-15T18:56:11+00:00 --- --- # Drill to Detail Ep.98 'Why the Future of Notebooks Isn't Just About Notebooks' with Special Guest Izzy Miller Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa361b012980c8170ce2b URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa361b012980c8170ce2b Description: Mark Rittman is joined by Izzy Miller to talk about Hex, a collaborative data platform that brings everyone together to explore, analyze, and share … and why the future of notebooks isn’t just about notebooks. Hex website Public app gallery of example Hex projects . Dataframe SQL announcement Hex fo Show: Drill to Detail Published: 2022-05-24T08:02:43+00:00 --- Mark Rittman is joined by Izzy Miller to talk about Hex, a collaborative data platform that brings everyone together to explore, analyze, and share … and why the future of notebooks isn’t just about notebooks. [Hex website](https://hex.tech/) [Public app gallery of example Hex projects](https://hex.tech/gallery/) . [Dataframe SQL announcement](https://hex.tech/blog/dataframe-sql) [Hex for Analytics Engineers](https://hex.tech/blog/hex-for-analytics-engineers-data-transformation) [Tweet thread about why the python / sql debate is boring](https://twitter.com/isidoremiller/status/1486764910943956993) --- # Drill to Detail Ep.97 'The Role of GA4 in the Future MarTech Stack' with Special Guest Daniel Perry-Reed Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa362f0ce0a7b298fb671 URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa362f0ce0a7b298fb671 Description: We’re joined on this episode of Drill to Detail by fellow Brightonian Daniel Perry-Reed from Measurelab, talking with Mark about the new Google Analytics 4 release and its role within the future Marketing Technology stack. From the wild west of data collection to conversion modelling The sun is sett Show: Drill to Detail Published: 2022-05-12T15:16:27+00:00 --- We’re joined on this episode of Drill to Detail by fellow Brightonian Daniel Perry-Reed from Measurelab, talking with Mark about the new Google Analytics 4 release and its role within the future Marketing Technology stack. [From the wild west of data collection to conversion modelling](https://www.measurelab.co.uk/blog/data-collection-conversion-modelling?utm_medium=podcast&utm_source=drill_to_detail) [The sun is setting on Universal Analytics](https://www.measurelab.co.uk/blog/sunsetting-universal-analytics-announcement?utm_medium=podcast&utm_source=drill_to_detail) [MeasureFest April 2022: The role of GA in the future MarTech stack](https://danalytics.co.uk/2022/04/06/measurefest-april-2022?utm_medium=podcast&utm_source=drill_to_detail) --- # Drill to Detail Ep.96 'Omni's Mission to Answer the First 100 Questions' with Special Guest Colin Zima Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3637ceb451a305cf9a7 URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3637ceb451a305cf9a7 Description: Mark Rittman is joined in this very special episode by Colin Zima, ex-Head of Analytics at Looker and now co-founder of Omni, a new analytics startup looking to become the fastest way to ask the first 100 questions. Introducing Omni: The fastest way to turn SQL into intelligence Omni Analytics Homep Show: Drill to Detail Published: 2022-04-26T14:29:10+00:00 --- Mark Rittman is joined in this very special episode by Colin Zima, ex-Head of Analytics at Looker and now co-founder of Omni, a new analytics startup looking to become the fastest way to ask the first 100 questions. [Introducing Omni: The fastest way to turn SQL into intelligence](https://www.linkedin.com/pulse/introducing-omni-fastest-way-turn-sql-intelligence-colin-zima-1c/) [Omni Analytics Homepage](https://www.exploreomni.com/) --- # Drill to Detail Ep.95 'Mozart Data and solving the Modern Data Stack Skills Shortage' with Special Guest Peter Fishman Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa36598f6c75bbfa8117a URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa36598f6c75bbfa8117a Description: Mark Rittman is joined in this episode by Peter Fishman to talk about Mozart Data, the modern data stack and the need to automate the role of the analytics engineer. MozartData website Data Bites: A Virtual Lunch and Learn with Powered by Fivetran, Snowflake and Mozart Data Launch HN: Mozart Data (Y Show: Drill to Detail Published: 2022-04-07T17:41:46+00:00 --- Mark Rittman is joined in this episode by Peter Fishman to talk about Mozart Data, the modern data stack and the need to automate the role of the analytics engineer. [MozartData](https://www.mozartdata.com/) website [Data Bites: A Virtual Lunch and Learn with Powered by Fivetran, Snowflake and Mozart Data](https://resources.fivetran.com/on-demand-webinars/data-bites-a-virtual-lunch-amp-learn-with-powered-by-fivetran-snowflake-amp-mozart-data) [Launch HN: Mozart Data (YC S20) – One-stop shop for a modern data pipeline](https://news.ycombinator.com/item?id=27182720) --- # Drill to Detail Ep.94 'Rudderstack and the Warehouse-First Customer Data Platform' with Special Guest Eric Dodds Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa366840d71402f9804e7 URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa366840d71402f9804e7 Description: Mark Rittman is joined by Eric Dodds to talk about Rudderstack's founding story and open-source roots, Segment compatibility and event-based pricing vs. MTUs and Rudderstack's "Warehouse-First" approach to building customer data platforms. Rudderstack's Data Stack : Deep Dive Introducing RudderStack Show: Drill to Detail Published: 2022-03-22T08:03:03+00:00 --- Mark Rittman is joined by Eric Dodds to talk about Rudderstack's founding story and open-source roots, Segment compatibility and event-based pricing vs. MTUs and Rudderstack's "Warehouse-First" approach to building customer data platforms. [Rudderstack's Data Stack : Deep Dive](https://www.rudderstack.com/blog/rudderstacks-data-stack-deep-dive/) [Introducing RudderStack Cloud: The Warehouse-First CDP for Developers](https://www.rudderstack.com/blog/introducing-rudderstack-cloud-the-warehouse-first-cdp-for-developers) [Rudderstack, Snowplow and Open-Source CDP Alternatives to Segment](https://rittmananalytics.com/blog/2022/2/20/rudderstack-snowplow-and-open-source-cdp-alternatives-to-segment) --- # Drill to Detail Ep.93 'Lightdash, Looker and dbt as the BI Tool Metrics Layer' with Special Guest Katie Hindson Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3676c29d92b91bca90a URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3676c29d92b91bca90a Description: In this first episode of a new series, Mark Rittman is joined by Katie Hindson to talk about Lightdash, a new open-source alternative to Looker that uses dbt as its metrics layer and semantic model. Lightdash.com What is the dbt Metrics Layer? Lightdash, Looker and dbt as the BI Tool Metrics Layer Show: Drill to Detail Published: 2022-03-09T17:47:12+00:00 --- In this first episode of a new series, Mark Rittman is joined by Katie Hindson to talk about Lightdash, a new open-source alternative to Looker that uses dbt as its metrics layer and semantic model. [Lightdash.com](https://www.lightdash.com/) [What is the dbt Metrics Layer?](https://docs.getdbt.com/docs/dbt-cloud/using-dbt-cloud/cloud-metrics-layer) [Lightdash, Looker and dbt as the BI Tool Metrics Layer](https://rittmananalytics.com/blog/2022/2/1/lightdash-looker-and-dbt-as-the-bi-tool-metrics-layer) --- # Drill to Detail Ep.92 'dbt, Spectacles and Making Testing Sexy' with Special Guests Dylan Baker and Josh Temple Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3685398b443a878c04e URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3685398b443a878c04e Description: We're joined on this episode of Drill to Detail by Dylan Baker and Josh Temple to talk about developing multi-layer modern data stack projects, Spectacles’ recently released integration with dbt and what's now possible with their new public API Extending Spectacles with API-Triggered Runs Synchroniz Show: Drill to Detail Published: 2021-06-23T23:20:09+00:00 --- We're joined on this episode of Drill to Detail by Dylan Baker and Josh Temple to talk about developing multi-layer modern data stack projects, Spectacles’ recently released integration with dbt and what's now possible with their new public API [Extending Spectacles with API-Triggered Runs](https://www.spectacles.dev/post/extending-spectacles-with-api-triggered-runs) [Synchronizing Looker and dbt™ with Spectacles](https://www.spectacles.dev/post/synchronizing-looker-and-dbt-with-spectacles) [Drill to Detail Ep.82 'Looker Development, Automated Testing and Spectacles' with Special Guest Josh Temple](https://rittmananalytics.com/drilltodetail/2020/6/3/drill-to-detail-ep82-looker-development-automated-testing-and-spectacles-with-special-guest-josh-temple) --- # Drill to Detail Ep.91 'GraphQL, Data Platform Engineers and Dagster' With Special Guest Nick Schrock Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa369c11a4e4fc5ef426d URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa369c11a4e4fc5ef426d Description: Mark Rittman is joined in this episode by Nick Schrock, ex-Facebook Engineer and now Founder of Elementl to talk about GraphQL, data platform engineers and @dagsterio, a data orchestrator for machine learning, analytics, and ETL. Introducing Dagster Dagster homepage Elementl homepage Orchestrating d Show: Drill to Detail Published: 2021-06-01T21:08:56+00:00 --- Mark Rittman is joined in this episode by Nick Schrock, ex-Facebook Engineer and now Founder of Elementl to talk about GraphQL, data platform engineers and @dagsterio, a data orchestrator for machine learning, analytics, and ETL. [Introducing Dagster](https://dagster.io/blog/introducing-dagster) [Dagster homepage](https://dagster.io/) [Elementl homepage](https://www.elementl.com/) [Orchestrating dbt with Dagster](https://www.getdbt.com/coalesce/agenda/orchestrating-dbt-with-dagster) [Moving past Airflow: Why Dagster is the next-generation data orchestrator](https://dagster.io/blog/dagster-airflow) --- # Drill to Detail Ep.90 'Apache Kafka, Real-Time Stream Processing and ksqlDB' with Special Guest Michael Drogalis Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa36be3c3c40ab0e521e3 URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa36be3c3c40ab0e521e3 Description: Mark Rittman is joined in this episode by Michale Drogalis, Product Manager at Confluent to talk about the history of Apache Kafka and real-time stream processing, the origins of kSQL and kSQLdb and the rationale and use-cases addressed by these products. ksqldDB product homepage Introducing ksqlDB Show: Drill to Detail Published: 2021-05-20T13:34:23+00:00 --- Mark Rittman is joined in this episode by Michale Drogalis, Product Manager at Confluent to talk about the history of Apache Kafka and real-time stream processing, the origins of kSQL and kSQLdb and the rationale and use-cases addressed by these products. [ksqldDB product homepage](https://www.confluent.io/product/ksql/) [Introducing ksqlDB Blog Post](https://www.confluent.io/blog/intro-to-ksqldb-sql-database-streaming) [Kafka Streams vs. ksqlDB Blog Post](https://www.confluent.co.uk/blog/kafka-streams-vs-ksqldb-compared) --- # Drill to Detail Ep.89 'Firebolt and the History of Cloud Data Warehousing' with Special Guest Eldad Farkash Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa36caad13923337ef70c URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa36caad13923337ef70c Description: Mark Rittman is joined by Eldad Farkash, Co-Founder and CEO at Firebolt to talk about SiSense and Panorama, the history of cloud data warehouses and how Firebolt's technology delivers query results 4-6000x faster than Snowflake, Redshift and AWS Athena. Snowflake’s Co-Founder Marcin Żukowski Reflect Show: Drill to Detail Published: 2021-04-28T11:04:00+00:00 --- Mark Rittman is joined by Eldad Farkash, Co-Founder and CEO at Firebolt to talk about SiSense and Panorama, the history of cloud data warehouses and how Firebolt's technology delivers query results 4-6000x faster than Snowflake, Redshift and AWS Athena. [Snowflake’s Co-Founder Marcin Żukowski Reflects On His Time At CWI](https://www.cwi.nl/news/blogs/snowflake2019s-co-founder-marcin-zukowski-reflects-on-his-time-at-cwi) [Centrum Wiskunde & Informatica & MonetDB](https://www.cwi.nl/research/groups/database-architectures) [Eldad Farkash, inventor of In-Chip technology, takes top honors with IT Software (Ind](https://www.prnewswire.com/news-releases/sisense-cto--co-founder-wins-world-technology-award-recognizing-major-contribution-to-big-data-analytics-technology-232515621.html) --- # Drill to Detail Ep.88 'Superset, Preset and the Future of Business Intelligence' with Special Guest Maxime Beauchemin Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa36d532acf22a4196dd4 URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa36d532acf22a4196dd4 Description: Maxime Beauchemin returns to the Drill to Detail Podcast and joins Mark Rittman to talk about what's new with Apache Airflow 2.0, the origin story for Apache Superset and now Preset.io, why the future of business intelligence is open source and news on Marquez, a reference implementation of the Open Show: Drill to Detail Published: 2021-04-12T17:06:20+00:00 --- Maxime Beauchemin returns to the Drill to Detail Podcast and joins Mark Rittman to talk about what's new with Apache Airflow 2.0, the origin story for Apache Superset and now Preset.io, why the future of business intelligence is open source and news on Marquez, a reference implementation of the OpenLineage open source metadata service for the collection, aggregation, and visualization of a data ecosystem’s metadata sponsored by WeWork. [The Rise of the Data Engineer](https://www.freecodecamp.org/news/the-rise-of-the-data-engineer-91be18f1e603/) [Drill to Detail Ep.26 'Airflow, Superset & The Rise of the Data Engineer' with Special Guest Maxime Beauchemin](https://rittmananalytics.com/drilltodetail/2017/5/15/drill-to-detail-ep26-airflow-superset-the-rise-of-the-data-engineer-with-special-guest-maxime-beauchemin) --- # Drill to Detail Ep.87 'Sunopsis, Fast Data and Oracle Data Integration's Data Mesh' with Special Guest Jeff Pollock Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3704e0b6e40edc64d3a URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3704e0b6e40edc64d3a Description: Mark Rittman is joined in this episode by Jeff Pollock, Vice President Product Development for Fast Data at Oracle, to talk about the history of Oracle Data Integration from Warehouse Builder and Sunopsis to the move to the cloud and today’s distributed “data mesh”, based around Oracle’s fast data p Show: Drill to Detail Published: 2021-03-29T17:36:19+00:00 --- Mark Rittman is joined in this episode by Jeff Pollock, Vice President Product Development for Fast Data at Oracle, to talk about the history of Oracle Data Integration from Warehouse Builder and Sunopsis to the move to the cloud and today’s distributed “data mesh”, based around Oracle’s fast data product, Oracle GoldenGate. [Technology Brief : Dynamic Data Fabric and Trusted Data Mesh using the Oracle GoldenGate Platform](https://oracle.com/a/ocom/docs/techbrief-enterprisedatameshandgoldengate.pdf) [Data Mesh Part 1: Future of Data Integration with a Deep Dive into GoldenGate, Kafka and Spark](https://youtube.com/watch?v=hVqmXH_DLHQ) [How to Move Beyond a Monolithic Data Lake to a Distributed Data Mesh](https://martinfowler.com/articles/data-monolith-to-mesh.html) --- # Drill to Detail Ep. 86 'Reverse ETL, Hightouch and CDW as CDP' with Special Guest Tejas Manohar Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa371d336f6184806c1e0 URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa371d336f6184806c1e0 Description: Mark Rittman is joined by Tejas Manohar from Hightouch to talk about the concept of "reverse ETL", his journey from working on Segment's Personas product to co-founding Hightouch and his recent guest post on the Fivetran Blog, "Why Your Customer Data Platform Should Be the Data Warehouse". Why Your Show: Drill to Detail Published: 2021-03-15T18:03:09+00:00 --- Mark Rittman is joined by Tejas Manohar from Hightouch to talk about the concept of "reverse ETL", his journey from working on Segment's Personas product to co-founding Hightouch and his recent guest post on the Fivetran Blog, "Why Your Customer Data Platform Should Be the Data Warehouse". [Why Your Customer Data Platform Should Be the Data Warehouse](https://fivetran.com/blog/customer-data-platform-data-warehouse) [Hightouch Ushers In The Era Of Operational Analytics](https://www.forbes.com/sites/frederickdaso/2021/03/04/hightouch-ushers-in-the-era-of-operational-analytics/?sh=1396c75a49e0) [Hightouch homepage](https://www.hightouch.io/) --- # Drill to Detail Ep.85 'Data Quality, Data Trust and Preventing Data Downtime' with Special Guest Barr Moses Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa372217d30538de790c5 URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa372217d30538de790c5 Description: The Drill to Detail Podcast returns for a new series with Mark joined by special guest Barr Moses to talk about data-driven customer success, data quality and the story behind her new data reliability startup Monte Carlo. Monte Carlo - Data reliability delivered dbt Data Source Freshness Delivering Show: Drill to Detail Published: 2021-03-03T23:12:04+00:00 --- The Drill to Detail Podcast returns for a new series with Mark joined by special guest Barr Moses to talk about data-driven customer success, data quality and the story behind her new data reliability startup Monte Carlo. [Monte Carlo - Data reliability delivered](https://www.montecarlodata.com/) [dbt Data Source Freshness](https://docs.getdbt.com/docs/dbt-cloud/using-dbt-cloud/cloud-snapshotting-source-freshness) [Delivering End-to-End Data Observability with Looker and Monte Carlo](https://www.montecarlodata.com/delivering-end-to-end-data-observability-with-looker-and-monte-carlo/) [Data Observability: How to Build Your Own Anom](https://www.montecarlodata.com/data-observability-in-practice-using-sql-part-ii-schema-lineage/) --- # Drill to Detail Ep.84 '2020 Review & Coalesce Special' with Special Guests Tristan Handy & Stewart Bryson Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa37370a62b3e6b885154 URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa37370a62b3e6b885154 Description: The Drill to Detail Podcast returns for an end-of-year and Coalesce 2020 special episode, with Mark Rittman joined by Tristan Handy from Fishtown Analytics along with Stewart Bryson from Red Pill Analytics. Coalesce 2020 Drill to Detail Ep.78 ‘Coronavirus, Shutdowns and the Data Analytics Industry’ Show: Drill to Detail Published: 2020-12-08T13:23:20+00:00 --- The Drill to Detail Podcast returns for an end-of-year and Coalesce 2020 special episode, with Mark Rittman joined by Tristan Handy from Fishtown Analytics along with Stewart Bryson from Red Pill Analytics. [Coalesce 2020](https://www.getdbt.com/coalesce) [Drill to Detail Ep.78 ‘Coronavirus, Shutdowns and the Data Analytics Industry’ with Special Guest Seth Rosen](https://rittmananalytics.com/drilltodetail/2020/4/8/drill-to-detail-ep78-coronavirus-shutdowns-and-the-data-analytics-industry-with-special-guest-seth-rosen) [Snowflake Stock Has More Than Tripled Since Its IPO. CEO Frank Slootman Explains What’s Next.](https://www.barrons.com/articles/snowflake-stock-has-more-than-tripled-since-its-ipo-ceo-frank-slootman-explains-whats-next-51607340601) --- # Drill to Detail Ep.83 ‘Abstracting, Extending and Automating Looker Development with Futuremodel’ with Special Guest Davis Clark Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa37571cab9171f85ae20 URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa37571cab9171f85ae20 Description: Mark Rittman is joined in this episode by Davis Clark, Founder at Futuremodel, to talk about democratizing Looker development by automating and abstracting the process of LookML development. Futuremodel Homepage Futuremodel Early Access Beta Program Show: Drill to Detail Published: 2020-07-28T23:39:58+00:00 --- Mark Rittman is joined in this episode by Davis Clark, Founder at Futuremodel, to talk about democratizing Looker development by automating and abstracting the process of LookML development. Futuremodel [Homepage](https://futuremodel.io/) Futuremodel [Early Access Beta Program](https://futuremodel.io/beta) --- # Drill to Detail Ep.82 'Looker Development, Automated Testing and Spectacles' with Special Guest Josh Temple Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3761b7b6a0977db2585 URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3761b7b6a0977db2585 Description: Mark Rittman is joined in this episode by Josh Temple, Analytics Engineer at Spotify to talk about analytics development, automated testing and Spectacles, an open-source tool and SaaS service that automatically tests your LookML to ensure Looker always runs smoothly for your users. Spectacles produ Show: Drill to Detail Published: 2020-06-04T11:54:00+00:00 --- Mark Rittman is joined in this episode by Josh Temple, Analytics Engineer at Spotify to talk about analytics development, automated testing and Spectacles, an open-source tool and SaaS service that automatically tests your LookML to ensure Looker always runs smoothly for your users. [Spectacles product homepage](https://www.spectacles.dev/) and early access program [Data Tests in Looker](https://docs.looker.com/reference/model-params/test) [Automated testing in the modern data warehouse](https://medium.com/@josh.temple/automated-testing-in-the-modern-data-warehouse-d5a251a866af) [#JOIN19 - Using Customized Open-Source Tools With LookML](https://info.looker.com/join-2019/extending-lookml-development-with-customized-toold) [Co](https://rittmananalytics.com/blog/2019/6/11/continuous-integration-feature-branches-and-automated-build-tests-using-dbtcloud) --- # Drill to Detail Ep.81 'Meltano, Singer Taps and Open-Source Data Pipelines' with Special Guest Douwe Maan Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa377cbeeba6f100836bf URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa377cbeeba6f100836bf Description: Mark Rittman is joined in this episode by GitLab.com Lead Developer Douwe Maan to discuss the history of the open-source Meltano project and its recent refocus on becoming the "glue" that creates open-source data pipelines, using plugin technologies such as Singer Taps and dbt transformations. Melta Show: Drill to Detail Published: 2020-05-19T21:23:42+00:00 --- Mark Rittman is joined in this episode by GitLab.com Lead Developer Douwe Maan to discuss the history of the open-source Meltano project and its recent refocus on becoming the "glue" that creates open-source data pipelines, using plugin technologies such as Singer Taps and dbt transformations. [Meltano homepage and quickstart demo](https://meltano.com/) [Hey, data teams - We're working on a tool just for you](https://about.gitlab.com/blog/2018/08/01/hey-data-teams-we-are-working-on-a-tool-just-for-you/) [Revisiting the Meltano strategy: a return to our roots](https://meltano.com/blog/2020/05/13/revisiting-the-meltano-strategy-a-return-to-our-roots/) [Why we are building an open source platform for ELT pipelines](https://meltano.com/blog/2020/05/13/why-we-are-building-an-open-source-platform-for-elt-pipelines/) [Singer -](https://www.singer.io/) --- # Drill to Detail Ep.80 'Data Architecture and Data Teams at Hubspot' with Special Guest James Densmore Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa378c0e3125bd003a61a URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa378c0e3125bd003a61a Description: Mark Rittman is joined in this episode by Hubspot's Director of Data Infrastructure James Densmore to talk about distributed and remote-friendly data teams, DevOps with dbt and Apache Airflow and career path options for data engineers. How should I structure my data team? A look inside HubSpot, Away Show: Drill to Detail Published: 2020-05-05T18:40:47+00:00 --- Mark Rittman is joined in this episode by Hubspot's Director of Data Infrastructure James Densmore to talk about distributed and remote-friendly data teams, DevOps with dbt and Apache Airflow and career path options for data engineers. [How should I structure my data team? A look inside HubSpot, Away, M.M. LaFleur, and more](https://blog.getdbt.com/data-team-structure-examples/) (dbt Blog) [Software Engineers do not Need to Become Managers to Thrive](https://www.dataliftoff.com/software-engineers-do-not-need-to-become-managers-to-thrive/) (Data Liftoff Blog) [The Misunderstood Data Engineer](https://www.dataliftoff.com/the-misunderstood-data-engineer/) (Data Liftoff Blog) [Modular ELT](https://www.dataliftoff.com/modular-elt/) (Data Liftoff Blog) --- # Drill to Detail Ep.79 'Scaling the Modern Data Analytics Stack' with Special Guests Drew Banin and Stewart Bryson Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3791e2211127a77e621 URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3791e2211127a77e621 Description: Mark Rittman is joined by special guests Drew Banin, co-founder of Fishtown Analytics and maintainer of dbt (data build tool) and Stewart Bryson, long-time friend of the show and CEO/Co-founder of Red Pill Analytics to talk about scaling modern data stack projects from startups to the enterprise; ho Show: Drill to Detail Published: 2020-04-20T21:25:28+00:00 --- Mark Rittman is joined by special guests Drew Banin, co-founder of Fishtown Analytics and maintainer of dbt (data build tool) and Stewart Bryson, long-time friend of the show and CEO/Co-founder of Red Pill Analytics to talk about scaling modern data stack projects from startups to the enterprise; how do you deal with data quality issues when there’s no central record of customers, how do we introduce data governance, enterprise requirements and meet the needs of enterprise architects and how do we scale concepts such as agile and analytics as engineering beyond our initial champion and data team? [The dbt Viewpoint](https://docs.getdbt.com/docs/about/viewpoint) [Fishtown Analytics](https://www.fishtownanalytics.com/) and [Drew Banin](https://www.linkedin.com/in/drewbanin) [Multi-Channel Marketi](https://rittmananalytics.com/blog/2020/2/8/multichannel-attribution-bigquery-dbt-looker-segment) --- # Drill to Detail Ep.78 ‘Coronavirus, Shutdowns and the Data Analytics Industry’ with Special Guest Seth Rosen Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa37b1a30443df8dbed71 URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa37b1a30443df8dbed71 Description: In this special episode of Drill to Detail Mark Rittman is joined by Seth Rosen, Co-Founder and Principal at Hashpath to talk about the impact of COVID-19 and the subsequent economic shutdown on the business of analytics and product startups, the work being done in the community to help find a cure Show: Drill to Detail Published: 2020-04-08T19:50:29+00:00 --- In this special episode of Drill to Detail Mark Rittman is joined by Seth Rosen, Co-Founder and Principal at Hashpath to talk about the impact of COVID-19 and the subsequent economic shutdown on the business of analytics and product startups, the work being done in the community to help find a cure and the role that data and analytics can have in accelerating the post-shutdown recovery. [“A COVID-19 Dashboard for Massachusetts using Looker and BigQuery”](https://www.hashpath.com/2020/04/a-covid-19-dashboard-for-massachusetts-using-looker-and-bigquery/) - Hashpath.com blog [“The Virus Survival Strategy for your Startup"](https://steveblank.com/2020/03/17/the-virus-survival-strategy-for-your-startup/) [“COVID-19: Implications for business”](https://www.mckinsey.com/business-functions/risk/our-insights/covid-19-implications-for-business) - McKinsey --- # Drill to Detail Ep.77 ' Keboola, Scaling Analytics and Winning the Looker Join Hackathon' with Special Guest Pavel Dolezal Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa37cd336f6184806c255 URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa37cd336f6184806c255 Description: We return with a new episode featuring Keboola CEO Pavel Dolezal on scaling analytics adoption beyond technical developers, the original story behind Keboola and how their team and friends won the recent Looker Join 2019 Hackathon event in San Francisco. Keboola homepage “DataOps is NOT Just DevOps Show: Drill to Detail Published: 2020-01-24T19:51:57+00:00 --- We return with a new episode featuring Keboola CEO Pavel Dolezal on scaling analytics adoption beyond technical developers, the original story behind Keboola and how their team and friends won the recent Looker Join 2019 Hackathon event in San Francisco. [Keboola](https://www.keboola.com/) homepage [“DataOps is NOT Just DevOps for Data”](https://medium.com/data-ops/dataops-is-not-just-devops-for-data-6e03083157b7) - Data Kitchen Blog - [Looker Brings the Data Community Together at JOIN 2019](https://www.businesswire.com/news/home/20190909005157/en/Looker-Brings-Data-Community-JOIN-2019) --- # Drill to Detail Ep.76 'Segment, Ecosystems and Customer Data Platforms' with Special Guest Calvin French-Owen Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa37f1e2211127a77e958 URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa37f1e2211127a77e958 Description: Mark Rittman is joined in the final Drill to Detail Podcast episode of the year by Calvin French-Owen, co-founder and CTO at Segment to talk about Segment's founding story, partner ecosystems and their new customer data platform service, Segment Personas. analytics.js: The hassle-free way to integra Show: Drill to Detail Published: 2019-12-17T00:00:04+00:00 --- Mark Rittman is joined in the final Drill to Detail Podcast episode of the year by Calvin French-Owen, co-founder and CTO at Segment to talk about Segment's founding story, partner ecosystems and their new customer data platform service, Segment Personas. [analytics.js: The hassle-free way to integrate analytics into any web application](https://github.com/segmentio/analytics.js/) (Github.com) [The Million Dollar Engineering Problem](https://segment.com/blog/the-million-dollar-eng-problem/) (Segment Blog) [Tracking Customer Email Marketing Interactions using Segment Personas and Connections](https://rittmananalytics.com/blog/2019/12/15/segmentcdpemailtracking?utm_campaign=personas_email_tracking&utm_medium=social&utm_source=podcast) (Rittman Analytics Blog) [Segment Personas](https://segment.com/product/personas/) product homepage --- # Drill to Detail Ep.75 'Bootstrapping an Early-Stage SaaS Analytics Consultancy' with Special Guest Olivier Dupuis Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3812506152079779fe9 URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3812506152079779fe9 Description: Mark Rittman is joined in this specially-extended episode of Drill to Detail by Olivier Dupuis, founder of Lantrn Analytics to share his experiences growing a product analytics business using today’s modern, modular SaaS analytics tools Lantrns Analytics Rittman Analytics "How it Works" - Lantrns An Show: Drill to Detail Published: 2019-11-26T23:49:22+00:00 --- Mark Rittman is joined in this specially-extended episode of Drill to Detail by Olivier Dupuis, founder of Lantrn Analytics to share his experiences growing a product analytics business using today’s modern, modular SaaS analytics tools [Lantrns Analytics](https://www.lantrns.co/) [Rittman Analytics](https://rittmananalytics.com/) ["How it Works"](https://www.lantrns.co/how-it-works/) - Lantrns Analytics ["How Rittman Analytics does Analytics: Modern BI Stack Operational Analytics using Looker, Stitch, dbt and Google BigQuery"](https://rittmananalytics.com/blog/2019/5/7/how-rittman-analytics) - Rittman Analytics Blog --- # Drill to Detail Ep.74 'Accelerating Oracle from Express to Analytic Views' with Special Guest Bud Endress Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3823dfaab173709b2f8 URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3823dfaab173709b2f8 Description: Mark Rittman is joined in this episode by Bud Endress, Director, Product Management at Oracle to talk about the evolution of Oracle's query acceleration and in-database OLAP features from the acquisition Express Server back in the 90's to today's Autonomous Data Warehouse and Analytic Views. Analyti Show: Drill to Detail Published: 2019-11-13T00:13:11+00:00 --- Mark Rittman is joined in this episode by Bud Endress, Director, Product Management at Oracle to talk about the evolution of Oracle's query acceleration and in-database OLAP features from the acquisition Express Server back in the 90's to today's Autonomous Data Warehouse and Analytic Views. [Analytic Views, Oracle Database 12.2 … and Oracle’s Reboot of OLAP now Everything’s In-Memory (and SQL)](https://www.linkedin.com/pulse/analytic-views-oracle-database-122-oracles-reboot-olap-mark-rittman/) [Oracle9i OLAP uncovered](https://searchoracle.techtarget.com/tip/Oracle9i-OLAP-uncovered) [Overview of Analytic Views](https://docs.oracle.com/en/database/oracle/oracle-database/12.2/dwhsg/overview-analytic-views.html#GUID-D384C4EE-1671-4F89-BC69-2D3133194869) --- # Drill to Detail Ep. 73 'Luck, Thinking Different and Designing Looker Data Platform' with Special Guest Colin Zima Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3830a112f1a291b4af7 URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3830a112f1a291b4af7 Description: The Drill to Detail Podcast returns for a new season with Mark Rittman joined in this episode by Colin Zima, VP of Product and Chief Analytics Officer at Looker to talk about luck, thinking differently and the deliberate design choices behind Looker Data Platform. JOIN 2018 - Colin Zima Product Keyn Show: Drill to Detail Published: 2019-10-28T23:10:45+00:00 --- The Drill to Detail Podcast returns for a new season with Mark Rittman joined in this episode by Colin Zima, VP of Product and Chief Analytics Officer at Looker to talk about luck, thinking differently and the deliberate design choices behind Looker Data Platform. [JOIN 2018 - Colin Zima Product Keynote - Part 1](https://info.looker.com/youtube-join-2018/join-2018-colin-zima-product-keynote-part-1) [GrowthHackers.com AMA with Colin Zima, Chief Analytics Officer & VP of Product at Looker](https://growthhackers.com/amas/live-mar-10-ama-with-colin-zima-chief-analytics-officer-vp-of-product-at-looker) [HotelTonight - Story of a Modern Data-Driven Business](https://looker.com/blog/hoteltonight-story-of-a-modern-data-driven-business) [Drill](https://drilltodetail.rittmananalytics.com/podcast/2019/3/4/drill-to-detail-ep60-a-deeper-look-into-looker-with-special-guest-lloyd-tabb) --- # Drill to Detail Ep.72 'Conversion Rate Optimization and Other CRAP' with Special Guest Bhav Patel Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa384f4ff9a57c18153dd URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa384f4ff9a57c18153dd Description: In this final Drill to Detail Episode before we take a break for the summer, Mark Rittman is joined by Bhav Patel, founder of the London Conversion Rate, Optimization and Product Analytics Meetup to talk about Conversion Rate Optimization, Experimentation and A/B Testing, Customer vs. Product Analyt Show: Drill to Detail Published: 2019-08-05T10:00:02+00:00 --- In this final Drill to Detail Episode before we take a break for the summer, Mark Rittman is joined by Bhav Patel, founder of the London Conversion Rate, Optimization and Product Analytics Meetup to talk about Conversion Rate Optimization, Experimentation and A/B Testing, Customer vs. Product Analytics, Attribution and Personalization ... and the story behind the CRAP Meetups. [CRAP Talks: CRO, Analytics and Product London](https://www.meetup.com/CRAP-Talks-CRO-Analytics-Product-London/) [CRAP Talks on Slack](https://join.slack.com/t/crap-talks/shared_invite/enQtMjI3MjM0OTkzMTA2LTA1NmE3MzBjMDFjMDUwOGEwNDdkNmRlNWZhY2YwZTgwZmVmYTVjMDVmYWVjNGFkYTQwZGEyZmVmNTdiOTRiMTM) [Analytics, BigQuery, Looker and How I Became an Internet Meme for 48 Hours (C](https://www.slideshare.net/markrittman/analytics-bigquery-looker-and-how-i-became-an-internet-meme-for-48-hours) --- # Drill to Detail Ep.71 'The Rise of Snowflake Data Warehouse' With Special Guest Kent Graziano Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa386c0e3125bd003a980 URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa386c0e3125bd003a980 Description: Mark Rittman is joined in this episode of Drill to Detail by returning guest Kent Graziano, Chief Technical Evangelist for Snowflake, to talk about Snowflake Data Warehouse's cloud-first architecture and recent product announcements at Snowflake Summit 2019. Try Snowflake Data Warehouse for free Wat Show: Drill to Detail Published: 2019-07-22T22:18:46+00:00 --- Mark Rittman is joined in this episode of Drill to Detail by returning guest Kent Graziano, Chief Technical Evangelist for Snowflake, to talk about Snowflake Data Warehouse's cloud-first architecture and recent product announcements at Snowflake Summit 2019. [Try Snowflake Data Warehouse for free](https://trial.snowflake.com/?utm_source=snowflake-evangelism&utm_medium=referral&utm_campaign=tech-evangelism-self-ser) [Watch Summit 2019 Keynotes](https://www.snowflake.com/summit/) [Snowflake Materialized Views: A Fast, Zero-Maintenance, Accurate Solution](https://www.snowflake.com/blog/snowflake-materialized-views-a-fast-zero-maintenance-accurate-solution/) [Modern Data Sharing: The Opportunities Are Endless](https://www.snowflake.com/blog/modern-data-sharing-opportunities-endless/) --- # Drill to Detail Ep.70 'Oracle Analytics, Luke Skywalker and the Remarkable Return of Enterprise Analytics' featuring Special Guest Bruno Aziza Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3874999467c00732992 URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3874999467c00732992 Description: Mark Rittman is joined by Bruno Aziza, Group Vice President, AI, Data Analytics & Cloud at Oracle to talk about the recently-updated product roadmap for Oracle Analytics, Oracle BI in the marketplace, recent acquisitions in the analytics marketplace and the recent Oracle Analytics Summit at Skywalke Show: Drill to Detail Published: 2019-07-08T22:03:17+00:00 --- Mark Rittman is joined by Bruno Aziza, Group Vice President, AI, Data Analytics & Cloud at Oracle to talk about the recently-updated product roadmap for Oracle Analytics, Oracle BI in the marketplace, recent acquisitions in the analytics marketplace and the recent Oracle Analytics Summit at Skywalker Ranch, California. [Oracle Analytics Summit](https://www.youtube.com/watch?v=eY0IPNqzsy4) [Oracle Analytics Summit Recap](https://www.youtube.com/watch?v=XHme3x2MuR4) [Oracle Analytics for Applications: Oracle Analytics Summit Product Tour](https://www.youtube.com/watch?v=bor_Hdzim84) [Oracle Analytics: Honing 18+ products down to a single brand](https://www.zdnet.com/article/oracle-analytics-honing-18-products-down-to-a-single-brand/) --- # Drill to Detail Ep.69 'Looker, Tableau and Consolidation in the BI Industry' featuring Special Guests Tristan Handy and Stewart Bryson Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3886c29d92b91bcacee URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3886c29d92b91bcacee Description: Mark Rittman, Founder and CEO of Rittman Analytics is joined in this special episode of Drill to Detail by returning guests Tristan Handy, Founder and CEO of Fishtown Analytics and Stewart Bryson, Founder and CEO of Red Pill Analytics to talk about the recent acquisitions of Looker by Google Cloud P Show: Drill to Detail Published: 2019-06-24T11:30:00+00:00 --- Mark Rittman, Founder and CEO of Rittman Analytics is joined in this special episode of Drill to Detail by returning guests Tristan Handy, Founder and CEO of Fishtown Analytics and Stewart Bryson, Founder and CEO of Red Pill Analytics to talk about the recent acquisitions of Looker by Google Cloud Platform and Tableau by Salesforce, the wider story about consolidation in the BI industry and whether the trend in analytics tools is towards enterprise features … or open source? [Five Thoughts on the Looker Acquisition by GCPCloud](https://twitter.com/markrittman/status/1136667377586921473) - Mark Rittman, Twitter Thread [A Wave of Acquisitions in Business Intelligence](https://blog.fishtownanalytics.com/a-wave-of-acquisitions-in-business-intelligence-93ef319089d8) - Tristan Handy, Fishtown Analytics Blog --- # Drill to Detail Ep.68 ‘Confluent, Event-First Thinking and Streaming Real-Time Analytics’ With Special Guests Robin Moffatt and Ricardo Ferreira and Special Host Stewart Bryson Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa389644f585f453a1b0a URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa389644f585f453a1b0a Description: In this special edition of the Drill to Detail Podcast hosted by Stewart Bryson, CEO and Co-Founder of Red Pill Analytics, he is joined by Robin Moffatt and Ricardo Ferreira, Developer Advocates at Confluent, to talk about Apache Kafka and Confluent, event-first thinking and streaming real-time anal Show: Drill to Detail Published: 2019-06-17T13:45:34+00:00 --- In this special edition of the Drill to Detail Podcast hosted by Stewart Bryson, CEO and Co-Founder of Red Pill Analytics, he is joined by Robin Moffatt and Ricardo Ferreira, Developer Advocates at Confluent, to talk about Apache Kafka and Confluent, event-first thinking and streaming real-time analytics. Confluent Download: [https://www.confluent.io/download/](https://www.confluent.io/download/) Demo: [https://github.com/confluentinc/cp-demo/](https://github.com/confluentinc/cp-demo/) Slack group: [http://cnfl.io/slack](http://cnfl.io/slack) Mailing list: [https://groups.google.com/forum/#!forum/confluent-platform](https://groups.google.com/forum/#!forum/confluent-platform) From Zero to Hero with Kafka Connect: [http://rmoff.dev/ksldn19l-kafka-connect-slides](http://rmoff.dev/ksldn19l-kafka-connect-slides) No M --- # Drill to Detail Ep.67 'Thinking Probabilistically and Scaling-Out the Modern BI Stack' with Special Guest Dylan Baker Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa38b87a9a167124f7fd5 URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa38b87a9a167124f7fd5 Description: Mark Rittman is joined by Dylan Baker, freelance analytics consultant, to talk about thinking probabalistically, analytics within venture-funded startups, devops and its role in scaling-out the modern BI stack. dbanalytics.co lookeraccess utility Annie Duke – Wanna Bet? – [Invest Like the Best, EP.1 Show: Drill to Detail Published: 2019-06-03T12:51:55+00:00 --- Mark Rittman is joined by Dylan Baker, freelance analytics consultant, to talk about thinking probabalistically, analytics within venture-funded startups, devops and its role in scaling-out the modern BI stack. [dbanalytics.co](https://dbanalytics.co/) [lookeraccess utility](https://github.com/dbanalyticsco/lookeraccess) [Annie Duke – Wanna Bet? – \[Invest Like the Best, EP.125\]](http://investorfieldguide.com/annie/) [London dbt Meetup #2](https://www.meetup.com/London-dbt-Meetup/events/261131728/) --- # Drill to Detail Ep.66 'ETL, Incorta and the Death of the Star Schema' with Special Guest Matthew Halliday Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa38cb012980c8170d253 URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa38cb012980c8170d253 Description: Mark Rittman is joined by Matthew Halliday to talk about the challenge of ETL and analytics on complex relational OLTP data models, previous attempts to solve these problems with products such as Oracle Essbase and Oracle E-Business Suite Extensions for Oracle Endeca and how those experiences, and o Show: Drill to Detail Published: 2019-05-27T20:35:09+00:00 --- Mark Rittman is joined by Matthew Halliday to talk about the challenge of ETL and analytics on complex relational OLTP data models, previous attempts to solve these problems with products such as Oracle Essbase and Oracle E-Business Suite Extensions for Oracle Endeca and how those experiences, and others, led to his current role as co-founder and VP of Products at Incorta. [The Death of the Star Schema: 3 Key Innovations Driving the Rapid Demise](http://go.incorta.com/2018.12.12WebinarTheDeathoftheStarSchema_LandingPage.html) [Accelerating Analytics with Direct Data Mapping](http://go.incorta.com/Direct-Data-Mapping.html?utm_medium=website-content&utm_source=Incorta&utm_term=na&utm_content=na&utm_campaign=2017-wp-accelerating-analytics) --- # Drill to Detail Ep.65 'Bootstrapping, Growth Hacking and Supermetrics Data Pipelines for Digital Marketers' with Special Guest Mikael Thuneberg Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa38d775bb936ffb06ecc URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa38d775bb936ffb06ecc Description: Mark Rittman is joined by CEO and Founder of Supermetrics, Mikael Thuneberg, to tell the story of how a mention in the official Google Analytics Blog and a prize of a t-shirt led to him founding and bootstrapping a €2M ARR marketing analytics business that’s probably the most important software vend Show: Drill to Detail Published: 2019-05-13T23:15:50+00:00 --- Mark Rittman is joined by CEO and Founder of Supermetrics, Mikael Thuneberg, to tell the story of how a mention in the official Google Analytics Blog and a prize of a t-shirt led to him founding and bootstrapping a €2M ARR marketing analytics business that’s probably the most important software vendor the Drill to Detail audience has never heard of, and who recently moved into the data pipelines-as-a-service market in collaboration with Google Cloud Platform and the Google BigQuery team. [Announcing Supermetrics for BigQuery: Get a marketing data warehouse up and running in minutes](https://supermetrics.com/blog/supermetrics-for-bigquery) (Supermetrics blog) [Supermetrics, Google BigQuery and Data Pipelines for Digital Marketers](https://rittmananalytics.com/blog/2019/4/14/supermetrics-google-bigquery-and-data-pipelines-for-digital-marketers) (Rittman Analytics blog) --- # Drill to Detail Ep.64 ‘Google BigQuery, BI Engine and the Future of Data Warehousing’ with Special Guest Jordan Tigani Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa38ee54d165bf7775b7a URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa38ee54d165bf7775b7a Description: Mark Rittman is joined in this episode by Jordan Tigani, Director of Product Management at Google for Google BigQuery, to talk about the history of BigQuery and its technology origins in Dremel; how BigQuery has evolved since its original release to handle a wide range of data warehouse workloads; a Show: Drill to Detail Published: 2019-04-29T21:51:45+00:00 --- Mark Rittman is joined in this episode by Jordan Tigani, Director of Product Management at Google for Google BigQuery, to talk about the history of BigQuery and its technology origins in Dremel; how BigQuery has evolved since its original release to handle a wide range of data warehouse workloads; and new features announced for BigQuery at Google Next’19 including BI Engine, Storage API, Connected Sheets and a range of new data connectors from Supermetrics and Fivetran. [Modern Data Warehousing with BigQuery (Cloud Next '19)](https://www.youtube.com/watch?v=eOQ3YJKgvHE) [Modern data warehousing with BigQuery: a Q&A with Engineering Director Jordan Tigani](https://cloud.google.com/blog/products/data-analytics/modern-data-warehousing-with-bigquery-a-qa-with-engineering-director-jordan-tigani) [Introduc](https://cloud.google.com/bi-engine/docs/overview) --- # Drill to Detail Ep.63 'Stitch, Talend and the Evolution of Data Pipelines' with Special Guest Jake Stein Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa38f3dfaab173709b36a URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa38f3dfaab173709b36a Description: Mark Rittman is joined by returning Special Guest Jake Stein, former co-founder and CEO of Stitch and now SVP of Stitch at Talend to talk about the evolution of the data pipeline-as-a-service, data catalogs and data governance and the vision behind Talend’s acquisition of Stitch. ”The Vision behind Show: Drill to Detail Published: 2019-04-15T21:20:34+00:00 --- Mark Rittman is joined by returning Special Guest Jake Stein, former co-founder and CEO of Stitch and now SVP of Stitch at Talend to talk about the evolution of the data pipeline-as-a-service, data catalogs and data governance and the vision behind Talend’s acquisition of Stitch. [”The Vision behind Talend's acquisition of Stitch”](https://www.stitchdata.com/blog/the-vision-behind-talends-acquisition-of-stitch/) [“Stitch is joining Talend”](https://www.stitchdata.com/blog/stitch-is-joining-talend/?utm_medium=blog&utm_campaign=eltalend) [“dbt: Analytics Engineering that Works”](https://www.getdbt.com/) [Talend](https://www.talend.com/) homepage --- # Drill to Detail Ep.62 'Enterprise analytics, DW Modernization and the rediscovery of Data Governance' With Special Guest Mike Ferguson Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3911a30443df8dbf3c7 URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3911a30443df8dbf3c7 Show: Drill to Detail Published: 2019-04-03T09:26:11+00:00 --- --- # Drill to Detail Ep.61 'Kaggle, DataRobot and How to Be a ... Zillionaire!' With Special Guest Jordan Meyer Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa392beee67022256da2b URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa392beee67022256da2b Description: Mark Rittman is joined by DataRobot’s Jordan Meyer to discuss Kaggle, machine learning, deep neural networks and his team’s strategy to win the $1m ZIllow Prize, beating over 1000 other teams to come up with the most accurate home value prediction. Zillow Prize: Zillow's Home Value Prediction (Zesti Show: Drill to Detail Published: 2019-03-20T14:15:47+00:00 --- Mark Rittman is joined by DataRobot’s Jordan Meyer to discuss Kaggle, machine learning, deep neural networks and his team’s strategy to win the $1m ZIllow Prize, beating over 1000 other teams to come up with the most accurate home value prediction. [Zillow Prize: Zillow's Home Value Prediction (Zestimate) | Kaggle](https://www.kaggle.com/c/zillow-prize-1) [And the Zillow Prize Goes to…Team ChaNJestimate!](https://www.zillow.com/promo/zillow-prize/) [Meet the ‘Zillow Prize’ winners who get $1M and bragging rights for beating the Zestimate](https://www.geekwire.com/2019/meet-zillow-prize-winners-get-1m-bragging-rights-zestimate-beating-algorithm/) [https://github.com/jordanmeyer](https://github.com/jordanmeyer) [Data Scientist Spotlight: Jordan Meyer](https://blog.datarobot.com/data-scientist-spotlight-jordan-meyer) --- # Drill to Detail Ep.60 'A Deeper Look Into Looker' With Special Guest Lloyd Tabb Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa39398f6c75bbfa81875 URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa39398f6c75bbfa81875 Description: The Drill to Detail Podcast returns for a new series with our host, Mark Rittman, joined by Lloyd Tabb, Founder CTO and Chairman of Looker to talk about the foundational story of Looker and LookML, query latency and semantic models, analytic engines and code IDEs, analytics developer workflows and t Show: Drill to Detail Published: 2019-03-05T12:02:28+00:00 --- The Drill to Detail Podcast returns for a new series with our host, Mark Rittman, joined by Lloyd Tabb, Founder CTO and Chairman of Looker to talk about the foundational story of Looker and LookML, query latency and semantic models, analytic engines and code IDEs, analytics developer workflows and the rise of cloud elastically-scalable databases, packaged applications and embedded analytics and why learning (and loving) are the long-term keys to analytics and business success. [Looker JOIN: The Tour 2019](https://looker.com/events/join-the-tour) [“7 Reasons Looker Built a New Language for Data”](https://looker.com/blog/7-reasons-looker-built-a-new-language-for-data) [“How do you decide what to model in dbt vs LookML?”](https://blog.fishtownanalytics.com/how-do-you-decide-what-to-model-in-dbt-vs-lookml-dca4c79e2304) - Tristan Handy --- # Drill to Detail Ep.59 'Looker, dbt and Digital Analytics Today' With Special Guests Tristan Handy and Stewart Bryson Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa394a663be77908a5c94 URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa394a663be77908a5c94 Description: Mark Rittman is joined in this Looker JOIN 2018 Special by long-term friends of the show Tristan Handy from Fishtown Analytics, and Stewart Bryson from Red Pill Analytics to talk about dbt and enabling data engineering for data analysts; the state of modern data analytics consulting today, and what Show: Drill to Detail Published: 2018-10-02T15:33:32+00:00 --- Mark Rittman is joined in this Looker JOIN 2018 Special by long-term friends of the show Tristan Handy from Fishtown Analytics, and Stewart Bryson from Red Pill Analytics to talk about dbt and enabling data engineering for data analysts; the state of modern data analytics consulting today, and what we’re looking forward to hearing about at next week’s Looker JOIN 2018 conference in San Francisco, CA. [Looker JOIN 2018, San Francisco](https://looker.com/events/join) [dbt (data build tool)](https://www.getdbt.com/) [“Future Proof your Analytics Stack”](https://about.modeanalytics.com/future-proof-stack/) --- # Drill to Detail Ep.58 'Candy Crush Saga, Analytics and Mobile Gaming' With Special Guest Jonathan Palmer Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa39571cab9171f85bdef URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa39571cab9171f85bdef Description: Mark Rittman is joined in this episode by Jonathan Palmer from King Games to talk about the role of analytics in the development of Candy Crush Saga and other King games, their use of Looker along with Google BigQuery and Exasol to provide analytics capabilities to their game designers and product o Show: Drill to Detail Published: 2018-07-03T14:30:50+00:00 --- Mark Rittman is joined in this episode by Jonathan Palmer from King Games to talk about the role of analytics in the development of Candy Crush Saga and other King games, their use of Looker along with Google BigQuery and Exasol to provide analytics capabilities to their game designers and product owners and his approach to doing all of this in a fast-moving, technology-driven internet business. \- [Candy Crush Saga](https://en.wikipedia.org/wiki/Candy_Crush_Saga) article on Wikipedia \- [King Games](https://king.com/) company website \- [“How King Games is Crushing Games Data”](https://www.dataiq.co.uk/article/how-king-crushing-games-data) DataIQ article \- [Looker and King Games case study](https://looker.com/customers/king) \- [Jonathan Palmer on LinkedIn](https://www.linkedin.com/in/jonathan-palmer-46ba6434/) --- # Drill to Detail Ep.57 'WhereScape Red and Data Warehouse Automation' With Special Guest Neil Barton Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3978a76676f10f3908b URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3978a76676f10f3908b Description: Mark Rittman is joined by Neil Barton, Chief Technology Officer at WhereScape to talk about metadata-driven data warehouse design, automating the build and management of data warehouse infrastructure and the thinking behind his company's WhereScape Red and WhereScape 3D tools. Show: Drill to Detail Published: 2018-06-18T23:01:34+00:00 --- Mark Rittman is joined by Neil Barton, Chief Technology Officer at WhereScape to talk about metadata-driven data warehouse design, automating the build and management of data warehouse infrastructure and the thinking behind his company's WhereScape Red and WhereScape 3D tools. --- # Drill to Detail Ep.56 'Essbase Cloud, Analytics Cloud and Cloud Analytics Consulting' with Special Guest Matt Yorke Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3985398b443a878d9c2 URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3985398b443a878d9c2 Description: In this specially-extended episode just before ODTUG KScope'18, Mark Rittman is joined by Matt Yorke from Qubix to talk about Oracle Essbase Cloud, Oracle Analytics Cloud and the business of Oracle Cloud analytics consulting Essbase Cloud is Here! Oracle Analytics Partner Forum 2018 Oracle Autonomou Show: Drill to Detail Published: 2018-06-04T15:43:15+00:00 --- In this specially-extended episode just before ODTUG KScope'18, Mark Rittman is joined by Matt Yorke from Qubix to talk about Oracle Essbase Cloud, Oracle Analytics Cloud and the business of Oracle Cloud analytics consulting [Essbase Cloud is Here!](http://www.qubix.com/blog/tech-talk/essbase-cloud-is-here) [Oracle Analytics Partner Forum 2018](https://mattyorke.com/2018/04/19/oracle-analytics-partner-forum-2018/) [Oracle Autonomous Analytics Cloud](https://www.oracle.com/solutions/business-analytics/analytics-cloud.html) [ODTUG KScope’18](https://kscope18.odtug.com/) [Matt Yorke on LinkedIn](https://www.linkedin.com/in/matt-yorke-6703332/) --- # Drill to Detail Ep.55 'Snowplow, Data Pipelines and Event-Level Digital Analytics' with Special Guest Yali Sassoon Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa39998f6c75bbfa81b7d URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa39998f6c75bbfa81b7d Description: Mark Rittman is joined by Yali Sassoon from Snowplow to talk about data pipelines and Hadoop in the cloud; how web analytics evolved from counting pageviews to today's event-level analysis of consumer behavoir across all digital channels; why digital analytics is hard but interesting; and Snowplow's Show: Drill to Detail Published: 2018-05-21T21:32:34+00:00 --- Mark Rittman is joined by Yali Sassoon from Snowplow to talk about data pipelines and Hadoop in the cloud; how web analytics evolved from counting pageviews to today's event-level analysis of consumer behavoir across all digital channels; why digital analytics is hard but interesting; and Snowplow's approach to building a successful hybrid open-source/commercial software business that competes successfully with megavendors such as Google and Adobe. [Snowplow website ](https://snowplowanalytics.com/) [Snowplow Insights](https://snowplowanalytics.com/products/snowplow-insights/) commercial hosted service details [Snowplow Open-Source ](https://snowplowanalytics.com/products/snowplow-open-source/) [Evolving Your Pipeline - Yali Sassoon - Snowplow Berlin Meetup #3](https://www.youtube.com/watch?v=3LM-FBPFWYo) [Snowplow on Looker](https://www.youtube.com/watch?v=8UKPKrHe3mM) --- # Drill to Detail Ep.54 'DataRobot & Machine Learning Automation' with Special Guest Greg Michaelson Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa39a1a30443df8dbf3ea URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa39a1a30443df8dbf3ea Description: Mark Rittman is joined in this episode by Greg Michaelson from DataRobot, talking about the benefits of automating the discovery and automation of analytics and machine learning in financial services and other industries. DataRobot: Automated Machine Learning for Predictive Modeling DataRobot for Bu Show: Drill to Detail Published: 2018-05-10T07:44:01+00:00 --- Mark Rittman is joined in this episode by Greg Michaelson from DataRobot, talking about the benefits of automating the discovery and automation of analytics and machine learning in financial services and other industries. [DataRobot: Automated Machine Learning for Predictive Modeling](https://www.datarobot.com/) [DataRobot for Business Analysts](https://www.datarobot.com/business-analysts/) [Automated Machine Learning Drives Intelligent Business](https://www.informationweek.com/big-data/ai-machine-learning/automated-machine-learning-drives-intelligent-business/a/d-id/1330249?) (Jen Underwood, Information Week article) [Marketing Attribution, Artificial Intelligence, and Game Theory](https://blog.datarobot.com/marketing-attribution-artificial-intelligence-and-game-theory) [Greg Michaelson](https://www.linkedin.com/in/greg-michaelson/) LinkedIn Profile --- # Drill to Detail Ep.53 'ThoughtSpot, Search and AI-Powered BI' With Special Guest Doug Bordonaro Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa39cf8c90e73dc28403a URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa39cf8c90e73dc28403a Description: Mark Rittman is joined by ThoughtSpot's Chief Data Evangelist Doug Bordonaro to talk about the value of data, issues around trust and consent raised by the EU's new GDPR regulations, and how ThoughtSpot are applying ideas from search engines combined with artificial intelligence smarts to surface in Show: Drill to Detail Published: 2018-04-23T23:03:06+00:00 --- Mark Rittman is joined by ThoughtSpot's Chief Data Evangelist Doug Bordonaro to talk about the value of data, issues around trust and consent raised by the EU's new GDPR regulations, and how ThoughtSpot are applying ideas from search engines combined with artificial intelligence smarts to surface insights and drive real value for business users from their analytics investment. [Value Becomes the 5th “V” in Big Data Factors](https://www.thoughtspot.com/fact-and-dimension/value-becomes-5th-v-big-data-factors) [ThoughtSpot - Search and AI-Driven Analytics for Humans](https://www.thoughtspot.com/) [The ThoughtSpot Blog](https://www.thoughtspot.com/blog) [Doug Bordonaro on LinkedIn](https://www.linkedin.com/in/dbordonaro/) [“Will GDPR Make Machine Learning Illegal?”](https://www.kdnuggets.com/2018/03/gdpr-machine-learning-illegal.html) --- # Drill to Detail Ep.52 'Lyft, Ride-Share Analytics and ETL Developer Productivity ' With Special Guest Mark Grover Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa39ef0ce0a7b298fbf3a URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa39ef0ce0a7b298fbf3a Description: Mark Grover LinkedIn Profile and Github Profile "Hadoop Application Architectures" "Drill to Detail Ep. 7 'Apache Spark and Hadoop Application Architectures' Lyft Engineering Blog "Software Engineer to Product Manager" blog by Gwen Shapira "Introduction to the Oracle Data Integrator Topology" from t Show: Drill to Detail Published: 2018-04-09T19:45:26+00:00 --- Mark Grover [LinkedIn Profile](https://www.linkedin.com/in/grovermark/) and [Github Profile](http://github.com/markgrover) ["Hadoop Application Architectures"](http://shop.oreilly.com/product/0636920033196.do)  ["Drill to Detail Ep. 7 'Apache Spark and Hadoop Application Architectures'](https://www.drilltodetail.com/podcast/2016/11/1/drill-to-detail-ep7-apache-spark-and-hadoop-application-architectures-with-special-guest-mark-grover) [Lyft Engineering Blog](https://eng.lyft.com/) ["Software Engineer to Product Manager" blog by Gwen Shapira](https://blog.usejournal.com/software-engineer-to-product-manager-5d1eddaa3678) ["Introduction to the Oracle Data Integrator Topology"](https://docs.oracle.com/middleware/1212/odi/ODIDG/setup_topology.htm#ODIDG171) from the Oracle Data Integrator docs site [Apache Airflow](https://airflow.apache.org/) and --- # Drill to Detail Ep.51 'Druid, Imply and OLAP Analysis on Event-Level Datasets' With Special Guest Fangjin Yang Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa39f8a76676f10f393ab URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa39f8a76676f10f393ab Description: Mark Rittman is joined by Special Guest Fangjin Yang to talk about the history of Druid, a high-performance, column-oriented, distributed data store originally developed by the team at Metamarkets to provide fast ad-hoc access to large amounts of event-level marketing data, and his work at Imply to Show: Drill to Detail Published: 2018-03-13T07:38:02+00:00 --- Mark Rittman is joined by Special Guest Fangjin Yang to talk about the history of Druid, a high-performance, column-oriented, distributed data store originally developed by the team at Metamarkets to provide fast ad-hoc access to large amounts of event-level marketing data, and his work at Imply to commercialise Druid and build a suite of supporting query and data management tools. [Druid](http://druid.io/) project homepage [Druid - A Real-Time Analytical Data Store](http://static.druid.io/docs/druid.pdf) (pdf) [Druid - Learning about the Druid Architecture](https://dev.to/rusrushal13/learning-about-the-druid-architecture-184c) [Imply.io](https://imply.io/) homepage [Druid, Imply and Looker 5 bring OLAP Analysis to BigQuery’s Data Warehouse](https://medium.com/mark-rittman/druid-imply-and-looker-5-bring-olap-analysis-to-bigquerys-data-warehouse-ae0cb5ffb3a4) --- # Drill to Detail Ep.50 'Agile BI, Karl Marx and Our Man from Moscow' With Special Guests Stewart Bryson and Alex Gorbachev Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3a11a30443df8dbf713 URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3a11a30443df8dbf713 Description: Mark Rittman is joined in this 50th Episode Special by our original guest on the first episode of Drill to Detail, Stewart Bryson, to talk about developing agile BI applications using FiveTran, SnowflakeDB and Looker and his recent work developing a BI solution for Google Play Marketing using Google Show: Drill to Detail Published: 2018-02-27T08:58:55+00:00 --- Mark Rittman is joined in this 50th Episode Special by our original guest on the first episode of Drill to Detail, Stewart Bryson, to talk about developing agile BI applications using FiveTran, SnowflakeDB and Looker and his recent work developing a BI solution for Google Play Marketing using Google Data Studio and Google Cloud Platform. We're also joined later in the show by Alex Gorbachev from Pythian, our mystery guest who Stewart then interviews flawlessly armed only with a set of questions given to him as the guest was unveiled ... though be sure to listen past the final closing music for the bonus out-takes. [#115 Google Play Marketing with Dom Elliott and Stewart Bryson](https://www.gcppodcast.com/post/episode-115-google-play-marketing-with-dom-elliott-and-stewart-bryson/) [The Next-Generation Jump Program](http://redpillanalytics.com/jump-program-new/) [The Data Sharehouse is Here](https://www.snowflake.net/data-sharehouse/) --- # Drill to Detail Ep.49 'Trifacta, Google Cloud Dataprep and Data Wranging for Data Engineers' With Special Guest Will Davis Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3a37ceb451a305d0759 URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3a37ceb451a305d0759 Description: Mark Rittman is joined by Will Davis from Trifacta to talk about the public beta of Google Cloud Dataprep, Trifacta's data wrangling platform and topics including metadata management, data quality and data management for big data and cloud data sources. Google Cloud Dataprep on Google Cloud Platform Show: Drill to Detail Published: 2018-02-05T23:41:57+00:00 --- Mark Rittman is joined by Will Davis from Trifacta to talk about the public beta of Google Cloud Dataprep, Trifacta's data wrangling platform and topics including metadata management, data quality and data management for big data and cloud data sources. [Google Cloud Dataprep](https://cloud.google.com/dataprep/) on Google Cloud Platform ["Google Cloud Dataprep: Spreadsheet-Style Data Wrangling Powered by Google Cloud Dataflow"](https://medium.com/mark-rittman/google-cloud-dataprep-spreadsheet-style-data-wrangling-powered-by-google-cloud-dataflow-a48c405d81c) ["A New Cloud-Based Data Prep Solution from Google & Trifacta"](https://www.trifacta.com/blog/data-preparation-solution-google-trifacta/) [Trifacta](https://www.trifacta.com/) website ["A Breakthrough Approach to Exploring and Preparing Data"](https://www.trifacta.com/products/why-trifacta/) --- # Drill to Detail Ep.48 'Mondrian OLAP, Apache Calcite and Database Dis-Aggregation' With Special Guest Julian Hyde Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3a41b7b6a0977db30a6 URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3a41b7b6a0977db30a6 Description: - Oracle Designer page on Oracle.com - Bitmap Index page on Wikipedia - Mondrian project page on Github - Mondrian OLAP Server page on Wikipedia - MultiDimensional eXpressions (MDX) page on Wikipedia - Julian Hyde blog - Apache Calcite project homepage - Apache Calcite Introduction and Overview deck Show: Drill to Detail Published: 2018-01-23T13:38:45+00:00 --- \- [Oracle Designer page on Oracle.com](http://www.oracle.com/technetwork/testcontent/9i-collateral-084901.html) \- [Bitmap Index page on Wikipedia](https://en.wikipedia.org/wiki/Bitmap_index) \- [Mondrian project page on Github](https://github.com/pentaho/mondrian) \- [Mondrian OLAP Server page on Wikipedia](https://en.wikipedia.org/wiki/Mondrian_OLAP_server) \- [MultiDimensional eXpressions (MDX) page on Wikipedia](https://en.wikipedia.org/wiki/MultiDimensional_eXpressions) \- [Julian Hyde blog](http://julianhyde.blogspot.co.uk/) \- [Apache Calcite project homepage](https://calcite.apache.org/) \- [Apache Calcite Introduction and Overview deck](https://www.slideshare.net/julianhyde/apache-calcite-overview) \- [Streaming SQL presentation at Apex Big Data World](https://www.youtube.com/watch?v=BsUUCw3Qn2o) --- # Drill to Detail Ep.47 'Business Analytics 2018 Predictive and Best-Practice Christmas & New Year Special' With Special Guest Christian Berg Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3a656da3c5f6971893d URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3a656da3c5f6971893d Description: Mark is joined by long-term industry veteran and friend Christian Berg to talk about surviving fifteen years as a contractor in analytics industry, changes he's seen in the market and in how project are approached, the value in getting involved in the community, and in a specially extended Christmas Show: Drill to Detail Published: 2017-12-26T14:22:49+00:00 --- Mark is joined by long-term industry veteran and friend Christian Berg to talk about surviving fifteen years as a contractor in analytics industry, changes he's seen in the market and in how project are approached, the value in getting involved in the community, and in a specially extended Christmas and New Year edition we look back at what was topical in 2017 and what are Christian's predictions for 2018 ... and appoint Christian as Head of our Best Practices Found on the Internet. --- # Drill to Detail Ep.46 'Market Trends and Findings from the BI Survey 17' With Special Guest Dr. Carsten Bange Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3a7b9e9127fe03eaf21 URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3a7b9e9127fe03eaf21 Description: Mark Rittman is joined in this episode of Drill to Detail by Dr. Carsten Bange from BARC to talk about findings from the recently completed BI Survey 17 including the continuing move to modern BI platforms and self-service desktop tools, analytics adoption trends and the increasing incorporation of Show: Drill to Detail Published: 2017-12-19T22:50:31+00:00 --- Mark Rittman is joined in this episode of Drill to Detail by Dr. Carsten Bange from BARC to talk about findings from the recently completed BI Survey 17 including the continuing move to modern BI platforms and self-service desktop tools, analytics adoption trends and the increasing incorporation of BI functionality within business applications, the surprising topicality of master data management and data governance ... and whatever happened to Nigel Pendse and his legendary OLAP Report? --- # Drill to Detail Ep.45 'Tellius, YellowFin and the State of AI in Analytics Today' With Special Guest Jen Underwood Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3a91a010925e83f44af URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3a91a010925e83f44af Description: Mark Rittman is joined in this episode by returning special guest Jen Underwood to talk about what's new and innovative in the BI and analytics industry right now, and how AI and machine learning are this year's data discovery and data visualization. Show: Drill to Detail Published: 2017-12-13T22:31:31+00:00 --- Mark Rittman is joined in this episode by returning special guest Jen Underwood to talk about what's new and innovative in the BI and analytics industry right now, and how AI and machine learning are this year's data discovery and data visualization. --- # Drill to Detail Ep.44 'Pandas, Apache Arrow and In-Memory Analytics' With Special Guest Wes McKinney Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3aa5398b443a878e31c URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3aa5398b443a878e31c Description: Mark is joined in this episode of Drill to Detail by Wes McKinney, to talk about the origins of the Python Pandas open-source package for data analysis and his subsequent work as a contributor to the Kudu (incubating) and Parquet projects within the Apache Software Foundation and Arrow, an in-memory Show: Drill to Detail Published: 2017-12-08T00:25:17+00:00 --- Mark is joined in this episode of Drill to Detail by Wes McKinney, to talk about the origins of the Python Pandas open-source package for data analysis and his subsequent work as a contributor to the Kudu (incubating) and Parquet projects within the Apache Software Foundation and Arrow, an in-memory data structure specification for use by engineers building data systems and the de-facto standard for columnar in-memory processing and interchange. --- # Drill to Detail Ep.43 'Oracle Analytics, Data Visualization Desktop 4.0 and The Art of Product Management' with Special Guest Mike Durran Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3ab65074d66fb0f8f13 URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3ab65074d66fb0f8f13 Description: Mark is joined by Mike Durran from the Oracle Analytics Product Management team in this UKOUG Tech’17 special to talk about his route into product management via the Oracle Discoverer BI tool, Oracle’s latest product in this space Oracle Data Visualization Desktop 4 and its new features, and Mike’s Show: Drill to Detail Published: 2017-11-30T14:56:27+00:00 --- Mark is joined by Mike Durran from the Oracle Analytics Product Management team in this UKOUG Tech’17 special to talk about his route into product management via the Oracle Discoverer BI tool, Oracle’s latest product in this space Oracle Data Visualization Desktop 4 and its new features, and Mike’s upcoming sessions at the UK Oracle User Group’s Tech’17 event next week in Birmingham, UK. --- # Drill to Detail Ep.42 'Evaluex, ML and Optimizing BigQuery & Athena' With Special Guest Avi Zloof Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3ad1a30443df8dbf774 URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3ad1a30443df8dbf774 Description: Mark is joined in this episode by Avi Zloof from Evaluex to talk about the new world of elastically-provisioned cloud-hosted analytic databases such as Google BigQuery and Amazon Athena, how their pricing model and vendor strategy differs from the traditional database vendors, and how machine learni Show: Drill to Detail Published: 2017-11-12T22:53:46+00:00 --- Mark is joined in this episode by Avi Zloof from Evaluex to talk about the new world of elastically-provisioned cloud-hosted analytic databases such as Google BigQuery and Amazon Athena, how their pricing model and vendor strategy differs from the traditional database vendors, and how machine learning can be used to automate performance tuning and optimize workloads in this new world of large-scale distributed query and storage. --- # Drill to Detail Ep.41 'Developing with Google BigQuery and Google Cloud Platform' With Special Guest Felipe Hoffa Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3ae4e0b6e40edc655fd URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3ae4e0b6e40edc655fd Show: Drill to Detail Published: 2017-10-30T17:21:44+00:00 --- --- # Drill to Detail Ep.40 'Fivetran's Middleware for SaaS Data' With Special Guest Taylor Brown Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3af7ceb451a305d0d83 URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3af7ceb451a305d0d83 Description: Mark Rittman is joined in this episode by Taylor Brown from Fivetran to talk about middleware for SaaS data, their focus on integrations with SaaS vendors and how this differentiates their offering, his thoughts on packaged analytic applications announced at the recent Looker Join conference ... and Show: Drill to Detail Published: 2017-10-16T08:10:39+00:00 --- Mark Rittman is joined in this episode by Taylor Brown from Fivetran to talk about middleware for SaaS data, their focus on integrations with SaaS vendors and how this differentiates their offering, his thoughts on packaged analytic applications announced at the recent Looker Join conference ... and where the name "FiveTran" came from. --- # Drill to Detail Ep.39 'Gluent Update and Offloading to the Cloud' with Special Guest Michael Rainey Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3b02c0f15496ef43222 URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3b02c0f15496ef43222 Show: Drill to Detail Published: 2017-10-03T19:22:21+00:00 --- --- # Drill to Detail Ep.38 'Oracle's Big Data Strategy Before OOW' with Special Guest Jean-Pierre Dijcks Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3b170a62b3e6b885c4c URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3b170a62b3e6b885c4c Description: Drill to Detail returns for a new season with special guest Jean-Pierre Dijcks, to talk about Oracle's Big Data Strategy now and in the past, thoughts on distributed query and storage in the cloud, and previewing themes and announcements to look forward to at the upcoming Oracle Open World 2017 even Show: Drill to Detail Published: 2017-09-19T20:08:31+00:00 --- Drill to Detail returns for a new season with special guest Jean-Pierre Dijcks, to talk about Oracle's Big Data Strategy now and in the past, thoughts on distributed query and storage in the cloud, and previewing themes and announcements to look forward to at the upcoming Oracle Open World 2017 event running in San Francisco next month. --- # Drill to Detail Ep.37 'How Essbase Won the OLAP Wars' With Special Guest Cameron Lackpour Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3b2b73abc1aeb4113ed URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3b2b73abc1aeb4113ed Description: Mark Rittman is joined in this Summer Special episode by none other than Cameron Lackpour, Essbase expert and Oracle ACE Director, to talk about why and how Essbase won the OLAP wars, how Essbase Server works and the role it now plays in Oracle Analytics Cloud and his involvement with user groups ov Show: Drill to Detail Published: 2017-08-01T07:51:02+00:00 --- Mark Rittman is joined in this Summer Special episode by none other than Cameron Lackpour, Essbase expert and Oracle ACE Director, to talk about why and how Essbase won the OLAP wars, how Essbase Server works and the role it now plays in Oracle Analytics Cloud and his involvement with user groups over the years. In this specially extended edition he also gives us his reading recommendations for while you're at the pool or, as he will be, out camping, and he also shares his predictions for what we'll hear from Oracle and the analytics industry when he, and Drill to Detail, returns in the autumn after a well-deserved summer break. --- # Drill to Detail Ep.36 'Marketing Analytics, AI and the Rise of the Omni-channel Consumer' With Special Guest Mark Madsen Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3b4b73abc1aeb4113f6 URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3b4b73abc1aeb4113f6 Description: Mark Rittman is joined by Industry Analyst Mark Madsen to talk about marketing analytics and the rise of the omni-channel consumer, the use of AI in analytics and personalization and what this all means for brands, for advertisers and for marketers. Show: Drill to Detail Published: 2017-07-24T18:02:00+00:00 --- Mark Rittman is joined by Industry Analyst Mark Madsen to talk about marketing analytics and the rise of the omni-channel consumer, the use of AI in analytics and personalization and what this all means for brands, for advertisers and for marketers. --- # Drill to Detail Ep.35 'Stitch Data, Singer and ETL for Data Engineers' With Special Guest Jake Stein Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3b59054ed45532f3bc4 URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3b59054ed45532f3bc4 Description: In this episode Mark is joined by Jake Stein to talk about Stitch Data and their ETL tool for data engineers, the new open-source project Singer and his experiences building a software startup that both partners and competes with the big cloud platform vendors. Show: Drill to Detail Published: 2017-07-17T22:41:36+00:00 --- In this episode Mark is joined by Jake Stein to talk about Stitch Data and their ETL tool for data engineers, the new open-source project Singer and his experiences building a software startup that both partners and competes with the big cloud platform vendors. --- # Drill to Detail Ep.34 'Qlik, MSAS and the Strategy of Data Analytics' with Special Guest Donald Farmer Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3b6250615207977a809 URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3b6250615207977a809 Description: Mark Rittman is joined by Donald Farmer to talk about his work at Microsoft on SQL Server Analysis Services and Integration Services, why he moved to Qlik and the challenges of evolving a BI product strategy from focusing on desktops to focusing on the enterprise, and some advice for customers, soft Show: Drill to Detail Published: 2017-07-12T15:09:57+00:00 --- Mark Rittman is joined by Donald Farmer to talk about his work at Microsoft on SQL Server Analysis Services and Integration Services, why he moved to Qlik and the challenges of evolving a BI product strategy from focusing on desktops to focusing on the enterprise, and some advice for customers, software vendors and partners working with data and analytics tools. --- # Drill to Detail Ep.33 'Building Out Analytics Functions in Startups' With Special Guest Tristan Handy Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3b7181a6a175e3192a9 URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3b7181a6a175e3192a9 Description: In this episode Mark is joined by Tristan Handy from Fishtown Analytics to talk about building-out analytics functions in high-growth startups and three related blog posts on this topic. Show: Drill to Detail Published: 2017-07-03T20:01:07+00:00 --- In this episode Mark is joined by Tristan Handy from Fishtown Analytics to talk about building-out analytics functions in high-growth startups and three related blog posts on this topic. --- # Drill to Detail Ep.32 'What Really Works in eCommerce' With Special Guest Will Browne Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3bad336f6184806ca7f URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3bad336f6184806ca7f Description: Mark is joined by Qubit colleague Will Browne to talk about his recent academic paper co-authored with Mike Swarbrick Jones on what techniques are most successful in influencing and retaining eCommerce customers, using analytics and statistical analysis-at-scale to analyze and understand 20 billion Show: Drill to Detail Published: 2017-06-26T16:44:35+00:00 --- Mark is joined by Qubit colleague Will Browne to talk about his recent academic paper co-authored with Mike Swarbrick Jones on what techniques are most successful in influencing and retaining eCommerce customers, using analytics and statistical analysis-at-scale to analyze and understand 20 billion user journeys held in Qubit's Google Cloud Platform-hosted Customer Data Store. --- # Drill to Detail Ep.31 'Dremel, Druid and Data Modeling on Google BigQuery' With Special Guest Dan McClary Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3bb5398b443a878e978 URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3bb5398b443a878e978 Description: Mark is joined by returning special guest Dan McClary to talk about data modeling and database design on distributed query engines such as Google BigQuery, the underlying Dremel technology and Capacitor storage format that enables this cloud distributed data warehouse-as-a-service platform to scale Show: Drill to Detail Published: 2017-06-19T21:55:26+00:00 --- Mark is joined by returning special guest Dan McClary to talk about data modeling and database design on distributed query engines such as Google BigQuery, the underlying Dremel technology and Capacitor storage format that enables this cloud distributed data warehouse-as-a-service platform to scale to petabyte-size tables spanning tens of thousands of servers, and techniques to optimize BigQuery table joins using nested fields, table partitioning and denormalization . --- # Drill to Detail Ep.30 "AI, ML & Oracle's Adaptive Intelligence Applications" with Special Guest Jack Berkowitz Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3bcf4ff9a57c18161d7 URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3bcf4ff9a57c18161d7 Description: Oracle's Jack Berkowitz joins Mark Rittman to talk about a new category of continuously adapting, self-learning applications being built-out by Oracle that use machine learning together with enterprise and third-party data to create a new generation of intelligent HR, CX, SCM and ERP SaaS apps. Show: Drill to Detail Published: 2017-06-13T09:04:18+00:00 --- Oracle's Jack Berkowitz joins Mark Rittman to talk about a new category of continuously adapting, self-learning applications being built-out by Oracle that use machine learning together with enterprise and third-party data to create a new generation of intelligent HR, CX, SCM and ERP SaaS apps. --- # Drill to Detail Ep.29 'New-World BI Development using BigQuery, Looker, Kakfa and Streamsets' With Special Guest Stewart Bryson Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3bd30aa241bf3868b47 URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3bd30aa241bf3868b47 Description: Stewart Bryson returns to the show to join Mark Rittman to discuss new-world BI and data warehousing development using Google BigQuery and Amazon Athena, Apache Kafka and StreamSets, and talks about his experiences with Looker, the cloud-native BI tool that brings semantic modeling and modern develo Show: Drill to Detail Published: 2017-06-05T22:44:02+00:00 --- Stewart Bryson returns to the show to join Mark Rittman to discuss new-world BI and data warehousing development using Google BigQuery and Amazon Athena, Apache Kafka and StreamSets, and talks about his experiences with Looker, the cloud-native BI tool that brings semantic modeling and modern development practices to the world of business intelligence. --- # Drill to Detail Ep.28 'Oracle Analytics, 12c and Data Visualization' with Special Guest Adrian Ward Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3bf6ceb586b9cb89457 URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3bf6ceb586b9cb89457 Description: Mark Rittman is joined in this episode by Independent Consultant Adrian Ward to talk about Oracle Business Analytics, Data Visualization, the BI Applications and his new book on Oracle Business Intelligence 12c. Show: Drill to Detail Published: 2017-05-30T07:22:31+00:00 --- Mark Rittman is joined in this episode by Independent Consultant Adrian Ward to talk about Oracle Business Analytics, Data Visualization, the BI Applications and his new book on Oracle Business Intelligence 12c. --- # Drill to Detail Ep.27 'Apache Kafka, Streaming Data Integration and Schema Registry' with Special Guest Gwen Shapira Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3c04269756938abcd63 URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3c04269756938abcd63 Description: Mark Rittman is joined by Gwen Shapira from Confluent to talk about Apache Kafka, streaming data integration and how it differs from batch-based, GUI-developed ETL development, the problem with architects, exactly-once processing and how data governance is coming to Kafka development with Confluent' Show: Drill to Detail Published: 2017-05-22T22:22:41+00:00 --- Mark Rittman is joined by Gwen Shapira from Confluent to talk about Apache Kafka, streaming data integration and how it differs from batch-based, GUI-developed ETL development, the problem with architects, exactly-once processing and how data governance is coming to Kafka development with Confluent's new schema registry server. --- # Drill to Detail Ep.26 'Airflow, Superset & The Rise of the Data Engineer' with Special Guest Maxime Beauchemin Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3c1f0ce0a7b298fc64d URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3c1f0ce0a7b298fc64d Description: Mark Rittman is joined by Maxime Beauchemin to talk about analytics and data integration at Airbnb, the Apache Airflow and Airbnb Superset open-source projects, and his recent Medium article on "The Rise of the Data Engineer" Show: Drill to Detail Published: 2017-05-15T22:21:44+00:00 --- Mark Rittman is joined by Maxime Beauchemin to talk about analytics and data integration at Airbnb, the Apache Airflow and Airbnb Superset open-source projects, and his recent Medium article on "The Rise of the Data Engineer" --- # Drill to Detail Ep.25 'SAP BusinessObjects Analytics, AI and Digital Transformation' With Special Guest Timo Elliott Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3c2a663be77908a6d16 URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3c2a663be77908a6d16 Description: Mark Rittman is joined by Timo Elliott, originally of Business Objects and now Innovation Evangelist for SAP, to talk about the origins of self-service BI with Business Objects' innovative "Universe" and the role analytics now plays within SAP; why analytics is the most important function within you Show: Drill to Detail Published: 2017-05-01T20:41:34+00:00 --- Mark Rittman is joined by Timo Elliott, originally of Business Objects and now Innovation Evangelist for SAP, to talk about the origins of self-service BI with Business Objects' innovative "Universe" and the role analytics now plays within SAP; why analytics is the most important function within your organization and why the vast majority of analytics is still reporting (which isn't so bad); and the role AI and other innovations will play in analytics going in the future. --- # Drill to Detail Ep.24 'Tom Sawyer Software, Graph + Spatial Analytics .... and why a Tweet goes Viral' With Special Guest Kevin Madden Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3c370a62b3e6b886327 URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3c370a62b3e6b886327 Description: Mark Rittman is joined by Kevin Madden and Josh Feingold to talk about graph + spatial analytics, Tom Sawyer Software ... and why a tweet goes viral Show: Drill to Detail Published: 2017-04-04T17:31:54+00:00 --- Mark Rittman is joined by Kevin Madden and Josh Feingold to talk about graph + spatial analytics, Tom Sawyer Software ... and why a tweet goes viral --- # Drill to Detail Ep.23 ‘Looker, BigQuery and Analytics on Big Data’ With Special Guest Daniel Mintz Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3c5532acf22a41979ae URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3c5532acf22a41979ae Description: Mark Rittman is joined by Daniel Mintz from Looker to talk about BI and analytics on Google BigQuery, data modelling on the new generation of cloud-based distributed-data warehousing platforms, and Looker's re-introduction of semantic models to big data analytics developers Show: Drill to Detail Published: 2017-03-28T12:24:03+00:00 --- Mark Rittman is joined by Daniel Mintz from Looker to talk about BI and analytics on Google BigQuery, data modelling on the new generation of cloud-based distributed-data warehousing platforms, and Looker's re-introduction of semantic models to big data analytics developers --- # Drill to Detail Ep.22 'SnapLogic's Enterprise Integration Cloud' With Special Guest Craig Stewart Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3c68f4ac97e1921339c URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3c68f4ac97e1921339c Description: Mark Rittman is joined by Craig Stewart to talk about application and data integration, ODI and Sunopsis, SnapLogic's approach to hybrid on-premise/cloud integration and the rise of data preparation and dataflow-based cloud integration tools. Show: Drill to Detail Published: 2017-03-20T23:45:03+00:00 --- Mark Rittman is joined by Craig Stewart to talk about application and data integration, ODI and Sunopsis, SnapLogic's approach to hybrid on-premise/cloud integration and the rise of data preparation and dataflow-based cloud integration tools. --- # Drill to Detail Ep.21 'The Fall, and Rise, of Microsoft BI' With Special Guest Chris Webb Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3c7fe54c97dbb9bf8f2 URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3c7fe54c97dbb9bf8f2 Description: Mark Rittman is joined by Independent Consultant Chris Webb to talk about MDX & DAX, MSAS and SQL Server and the fall ... and rise, of Microsoft BI Show: Drill to Detail Published: 2017-03-14T17:24:09+00:00 --- Mark Rittman is joined by Independent Consultant Chris Webb to talk about MDX & DAX, MSAS and SQL Server and the fall ... and rise, of Microsoft BI --- # Drill to Detail Ep.20 'MapR Platform Differentation, Scaling Hadoop and Microservices' With Special Guest Tugdall Grall Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3c883fd5e4008aa770a URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3c883fd5e4008aa770a Description: Mark Rittman is joined in this episode by MapR's Tugdall Grall to talk about MapR's platform differentation and relationship with open-source Hadoop, scaling and streaming, microservices, and MapR's platform strategy around big data workloads in the cloud. Show: Drill to Detail Published: 2017-03-06T23:35:01+00:00 --- Mark Rittman is joined in this episode by MapR's Tugdall Grall to talk about MapR's platform differentation and relationship with open-source Hadoop, scaling and streaming, microservices, and MapR's platform strategy around big data workloads in the cloud. --- # Drill to Detail Ep.19 'Elasticsearch, Elastic Stack & Elastic Cloud' With Special Guest Mark Walkom Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3ca250615207977abc9 URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3ca250615207977abc9 Description: Mark Rittman is joined by Elastic's Mark Walkom to talk about Elasticsearch, Kibana, Logstash and the Elastic Stack; business models built-around an open-source software core; and their move into cloud services with Elastic Cloud Show: Drill to Detail Published: 2017-02-21T23:58:29+00:00 --- Mark Rittman is joined by Elastic's Mark Walkom to talk about Elasticsearch, Kibana, Logstash and the Elastic Stack; business models built-around an open-source software core; and their move into cloud services with Elastic Cloud --- # Drill to Detail Ep.18 'Oracle's Cloud Analytics and Data Visualization Strategy' With Special Guest Vasu Murthy Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3cb8a76676f10f39ad3 URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3cb8a76676f10f39ad3 Description: Mark Rittman is joined by Vasu Murthy, Oracle's Senior Director for Product Management of Oracle Business Analytics to talk about what's new with OBIEE and Oracle Data Visualization and the recently released Oracle Analytics Cloud, a dive into the technical architecture of these new additions to Ora Show: Drill to Detail Published: 2017-02-08T22:41:20+00:00 --- Mark Rittman is joined by Vasu Murthy, Oracle's Senior Director for Product Management of Oracle Business Analytics to talk about what's new with OBIEE and Oracle Data Visualization and the recently released Oracle Analytics Cloud, a dive into the technical architecture of these new additions to Oracle's BI platform, and Oracle's vision for hybrid on-prem/cloud analytics. --- # Drill to Detail Ep.17 ‘Mode 1 Analytics and the Future of Cloud DWs’ With Special Guest Rick Greenwald Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3cce3c3c40ab0e52db2 URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3cce3c3c40ab0e52db2 Description: Mark Rittman is joined by Gartner analyst and ex-Oracle Database Cloud Service PM Rick Greenwald to talk about IT’s continuing (and essential!) role in corporate BI&DW deployments and the debate around Mode1 vs. Mode2 Analytics, how we got here, and the future of data warehousing database platforms Show: Drill to Detail Published: 2017-02-02T17:06:44+00:00 --- Mark Rittman is joined by Gartner analyst and ex-Oracle Database Cloud Service PM Rick Greenwald to talk about IT’s continuing (and essential!) role in corporate BI&DW deployments and the debate around Mode1 vs. Mode2 Analytics, how we got here, and the future of data warehousing database platforms as we move into the cloud. --- # Drill to Detail Ep.16 'Qubit, Visitor Cloud & Google BigQuery' With Special Guest Alex Olivier Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3cd30aa241bf3868bc4 URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3cd30aa241bf3868bc4 Description: Mark Rittman is joined by Alex Olivier from Qubit to talk about their platform journey from on-premise Hadoop to petabytes of data running in Google Cloud Platform, using Google Cloud Dataflow (aka Apache Beam), Google PubSub and Google BigQuery along with machine learning and analytics to deliver p Show: Drill to Detail Published: 2017-01-24T08:18:21+00:00 --- Mark Rittman is joined by Alex Olivier from Qubit to talk about their platform journey from on-premise Hadoop to petabytes of data running in Google Cloud Platform, using Google Cloud Dataflow (aka Apache Beam), Google PubSub and Google BigQuery along with machine learning and analytics to deliver personalisation at-scale for digital retailers around the world. --- # Drill to Detail Ep.15 ‘Modern BI and the Gartner BI Magic Quadrant 2016’ With Special Guest Cindi Howson Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3ce840d71402f981c56 URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3ce840d71402f981c56 Description: Mark Rittman is joined in this episode by Cindi Howson to talk about Modern BI and last year's Gartner BI & Analytics Magic Quadrant, the rise of self-service BI and IT's new role as enabler rather than owner, CIO priorities around BI & Analytics ... and, is the BICC dead? Show: Drill to Detail Published: 2017-01-17T16:00:50+00:00 --- Mark Rittman is joined in this episode by Cindi Howson to talk about Modern BI and last year's Gartner BI & Analytics Magic Quadrant, the rise of self-service BI and IT's new role as enabler rather than owner, CIO priorities around BI & Analytics ... and, is the BICC dead? --- # Drill to Detail Ep.14 ‘Cloudy Big Data Paradigm-Shifting Christmas & New Year Special’ With Special Guest Robin Moffatt Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3d0840d71402f981c71 URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3d0840d71402f981c71 Description: Mark Rittman is joined in this Christmas & New Year special episode by none-other than Robin Moffatt, head of R&D at Rittman Mead and an old colleague from my consulting days, to talk about his experiences with Amazon Elastic MapReduce (EMR) for BI + Analytics and how "the devil's in the detail", an Show: Drill to Detail Published: 2016-12-21T10:00:10+00:00 --- Mark Rittman is joined in this Christmas & New Year special episode by none-other than Robin Moffatt, head of R&D at Rittman Mead and an old colleague from my consulting days, to talk about his experiences with Amazon Elastic MapReduce (EMR) for BI + Analytics and how "the devil's in the detail", and to hear his take on what happened in the BI and analytics world in 2016 and what to look out for in 2017. --- # Drill to Detail Ep.13 ‘Apache Drill, MapR + Bringing Data Discovery to Hadoop & NoSQL’ with Special Guest Neeraja Rentachintala Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3d1f0ce0a7b298fc9b8 URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3d1f0ce0a7b298fc9b8 Description: Mark Rittman is joined by MapR's Neeraja Rentachintala to talk about Apache Drill, Apache Arrow, MapR-DB, extending Hadoop-based data discovery to self-describing file formats and NoSQL databases, and why MapR backed Drill as their strategic SQL-on-Hadoop platform technology. Show: Drill to Detail Published: 2016-12-13T15:39:09+00:00 --- Mark Rittman is joined by MapR's Neeraja Rentachintala to talk about Apache Drill, Apache Arrow, MapR-DB, extending Hadoop-based data discovery to self-describing file formats and NoSQL databases, and why MapR backed Drill as their strategic SQL-on-Hadoop platform technology. --- # Drill to Detail Ep.12 'Gluent and the New World of Hybrid Data' with Special Guest Tanel Poder Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3d265074d66fb0f9c15 URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3d265074d66fb0f9c15 Description: Mark Rittman is joined by Gluent's Tanel Poder to talk about Hadoop, Gluent Data Platform, the coming of the hybrid world and how Hadoop will evolve as it moves into the cloud. Show: Drill to Detail Published: 2016-12-06T06:44:45+00:00 --- Mark Rittman is joined by Gluent's Tanel Poder to talk about Hadoop, Gluent Data Platform, the coming of the hybrid world and how Hadoop will evolve as it moves into the cloud. --- # Drill to Detail Ep.11 'OLAP, Consulting & BI Past Present + Future' with Special Guest Graham Spicer Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3d3beee67022256edb1 URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3d3beee67022256edb1 Description: Mark Rittman is joined by Graham Spicer to talk about BI past, present and future including the history of Oracle OLAP, running and owning a consulting business over 30 years of changes in the industry, what's hot on the market today, and what skills and techniques are still relevant as we adopt big Show: Drill to Detail Published: 2016-11-29T09:58:01+00:00 --- Mark Rittman is joined by Graham Spicer to talk about BI past, present and future including the history of Oracle OLAP, running and owning a consulting business over 30 years of changes in the industry, what's hot on the market today, and what skills and techniques are still relevant as we adopt big data products and new methods of analysis --- # Drill to Detail Ep.10 'Oracle's Big Data Reboot, and Data Storytelling' With Special Guest Stewart Bryson Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3d4532acf22a4197d43 URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3d4532acf22a4197d43 Description: Mark Rittman is joined once more by Stewart Bryson, talking about Oracle's recent reboot of it's cloud big data platform at Oracle Openworld 2016, thoughts on DataFlowML and comparisons with Google's Cloud DataFlow and Amazon Kinesis, and data storytelling with Oracle Data Visualisation Desktop 2.0 Show: Drill to Detail Published: 2016-11-22T18:29:22+00:00 --- Mark Rittman is joined once more by Stewart Bryson, talking about Oracle's recent reboot of it's cloud big data platform at Oracle Openworld 2016, thoughts on DataFlowML and comparisons with Google's Cloud DataFlow and Amazon Kinesis, and data storytelling with Oracle Data Visualisation Desktop 2.0 --- # Drill to Detail Ep.9 'Streamsets, Data-in-Motion and Data Drift' with Special Guest Pat Patterson Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3d6444e392bb4bc009b URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3d6444e392bb4bc009b Description: Mark Rittman is joined by StreamSets' Pat Patterson, talking about data in motion and doing it at scale, the story behind StreamSets and the problem of data drift, and the challenges involved in managing dataflows at scale as a continuous operation Show: Drill to Detail Published: 2016-11-15T15:30:45+00:00 --- Mark Rittman is joined by StreamSets' Pat Patterson, talking about data in motion and doing it at scale, the story behind StreamSets and the problem of data drift, and the challenges involved in managing dataflows at scale as a continuous operation --- # Drill to Detail Ep.8 'Self-Service BI, Data Prep & Big Data Vendor Strategy' with Special Guest Jen Underwood Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3d7aad13923337f1e03 URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3d7aad13923337f1e03 Description: Mark Rittman is joined by Jen Underwood to discuss the aftermath of the Gartner BI&A Magic Quadrant 2016 and the rise of self-service, Mode-2 analytics; innovation in predictive analytics and data preparation tools, and how the big data cloud vendors are differentiating themselves (or not) Show: Drill to Detail Published: 2016-11-08T06:31:52+00:00 --- Mark Rittman is joined by Jen Underwood to discuss the aftermath of the Gartner BI&A Magic Quadrant 2016 and the rise of self-service, Mode-2 analytics; innovation in predictive analytics and data preparation tools, and how the big data cloud vendors are differentiating themselves (or not) --- # Drill to Detail Ep.7 'Apache Spark, and Hadoop Application Architectures' with Special Guest Mark Grover Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3d8fe54c97dbb9bf95d URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3d8fe54c97dbb9bf95d Description: Mark is joined by Cloudera's Mark Grover to talk about his work with Apache Spark and Apache Spot, and the book "Hadoop Application Architectures" co-authored with Gwen Shapira, Jonathan Seidman and Ted Malaska Show: Drill to Detail Published: 2016-11-01T15:58:08+00:00 --- Mark is joined by Cloudera's Mark Grover to talk about his work with Apache Spark and Apache Spot, and the book "Hadoop Application Architectures" co-authored with Gwen Shapira, Jonathan Seidman and Ted Malaska --- # Drill to Detail Ep.6 'Data Capital and the Economics of Big Data' with Special Guest Paul Sonderegger Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3d9840d71402f981cb1 URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3d9840d71402f981cb1 Description: Mark Rittman is joined by special guest Paul Sonderegger, Oracle's Big Data Strategist, to talk about Data Capital and the economic impact of big data Show: Drill to Detail Published: 2016-10-25T10:59:32+00:00 --- Mark Rittman is joined by special guest Paul Sonderegger, Oracle's Big Data Strategist, to talk about Data Capital and the economic impact of big data --- # Drill to Detail Ep. 5 'SnowflakeDB, and is Data Modeling Dead?' with Special Guest Kent Graziano Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3db45ac16627dfe9930 URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3db45ac16627dfe9930 Description: In this episode Mark Rittman is joined by Kent Graziano to answer the questions "Is data modeling now dead", and "is SnowflakeDB the future of analytic data engines in the cloud?" Better Data Modeling: An Introduction to Agile Data Engineering Using Data Vault 2.0 Better Data Modeling: Tips for Enha Show: Drill to Detail Published: 2016-10-18T14:43:51+00:00 --- In this episode Mark Rittman is joined by Kent Graziano to answer the questions "Is data modeling now dead", and "is SnowflakeDB the future of analytic data engines in the cloud?" [Better Data Modeling: An Introduction to Agile Data Engineering Using Data Vault 2.0](http://www.amazon.com/Better-Data-Modeling-Introduction-Engineering-ebook/dp/B018BREV1C/) [Better Data Modeling: Tips for Enhancing Your Use of Oracle SQL Developer Data Modeler](http://www.amazon.com/Better-Data-Modeling-Enhancing-Developer-ebook/dp/B00UK75LYI/) [Kent Graziano Blog](https://kentgraziano.com/) --- # Drill to Detail Ep.4 'Reference Architectures Revisited' with Special Guest Andrew Bond Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3dcb012980c8170e63c URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3dcb012980c8170e63c Description: Mark Rittman is joined in this episode by Oracle's Andrew Bond, to talk about their Big Data Reference Architecture two years on" Show: Drill to Detail Published: 2016-10-11T11:06:00+00:00 --- Mark Rittman is joined in this episode by Oracle's Andrew Bond, to talk about their Big Data Reference Architecture two years on" --- # Drill to Detail Ep.3 'Apache Kudu and Cloudera's Analytic Platform' with Special Guest Mike Percy Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3dd8f4ac97e19213758 URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3dd8f4ac97e19213758 Description: Mark Rittman is joined by Cloudera's Mike Percy, to talk about Apache Kudu, Analytics on Hadoop, and Cloudera's work in this area Show: Drill to Detail Published: 2016-10-04T20:46:00+00:00 --- Mark Rittman is joined by Cloudera's Mike Percy, to talk about Apache Kudu, Analytics on Hadoop, and Cloudera's work in this area --- # Drill to Detail Ep.2. 'Future of SQL on Hadoop', With Special Guest Dan McClary Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3de56da3c5f69719fb9 URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3de56da3c5f69719fb9 Show: Drill to Detail Published: 2016-09-27T21:13:00+00:00 --- --- # Drill To Detail Ep.1. 'After The Gartner BI&A Magic Quadrant 2016', With Special Guest Stewart Bryson Canonical: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3dff4ff9a57c181660b URL: https://rittmananalytics.com/drilltodetail/63b6c9e3cbf92e7d56375d12%3A63eaa35fc9db0f39f7df748d%3A63eaa3dff4ff9a57c181660b Show: Drill to Detail Published: 2016-09-20T08:00:00+00:00 ---