Posts in Modern Analytics Stack
HubSpot Data Actions, Harvest Analytical Workflows and Looker Data Platform

At Rittman Analytics we use a number of SaaS applications to run our back office operations including HubSpot CRM for sales tracking, Harvest for timesheets and resource planning, Jira for project management and Xero for invoicing and accounting. Back in May 2019 I blogged about how we used services from Stitch, Fivetran and dbt to extract, transform and combine data from those services into a data warehouse running on Google BigQuery and since then we’ve built-out a number of finance, operations and customer success dashboards that provide our back office team with valuable insights into the performance of our business.

dashboards.png

But Looker is more than just an ad-hoc query and dashboarding tool, it’s a full data platform that integrates with SaaS applications such as Harvest, HubSpot, Xero and Salesforce at multiple levels:

  • At a data-source level, querying and analyzing data from those application sources

  • Within the Looker explore UI using features such as links, data actions and the Action Hub

  • Within business workflows, providing analytical insights and trusted metrics through the Looker API

Adding Links to HubSpot Deal and Company Pages as Explore Menu Items

One example of this is how we’ve added menu options in the Looker explore to provide direct links into HubSpot for viewing the details of a deal or a company within that application, and for our sales manager to request an update on a deal where there’s been no contact from either side for a while.

hubspot1.png

This type of simple integration is done through the “link” property in LookML that you can add to dimension and measure definitions, as I’ve done in the example below for our HubSpot opportunity name dimension.

dimension: opportunity_name {
    type: string
    sql: $.dealname ;;
    link: {
      label: "View Deal in Hubspot"
      url: "https://app.hubspot.com/contacts/4402794/deal/{  }/"
      icon_url: "http://app.hubspot.com/favicon.ico"
    }
}

Then when a user clicks on a field in an explore or a dashboard tile they’ll be able to click-through to HubSpot to view all of the context of the deal; if like us you use Google Apps authentication for both Looker and our SaaS applications you’ll be taken straight into HubSpot without having to login first.

link.png

Creating Looker Data Actions to Update HubSpot Deal Properties

A more advanced form of integration between Looker and HubSpot that we’ve setup is the ability for users to update details of a HubSpot deal directly from within Looker, a good example of how you can make your Looker content more “actionable”.

data_actions.gif

This type of integration involves another type of LookML dimension and measure property called an “action”. I first talked about actions in a blog post titled “Using Looker Data Actions to Make Monzo Spend Analysis More Interactive … and Actionable” back in 2018 where I used the feature to call services such as Google Maps, Google My Business and via Zapier, Google Tasks; in the case of our HubSpot integration we’ve again use Zapier for the webhook service that our data actions call, setting up a three stage Zapier “Zap” that takes the details of the deal and the values that need changing from the webhook payload, then searches for the deal in HubSpot that needs updating and then applies those changes to the deal record.

dealstage.png

Adding Data Action Menu Items to Request HubSpot Deal Updates via GMail

Another example of this type of data action integration we’ve setup is between Looker, HubSpot and GMail, giving us the ability to send an email to request a status update from the team directly from the Looker dashboard.

dbm.png

Leveraging the Looker API to Provide KPIs for Harvest Workflows Automated using Zapier

As well as providing hooks into HubSpot for managing the deal process directly from within Looker, we also use Looker to inform and provide trusted business key performance indicators (KPIs) for our automated workflows. An example of this is when we’ve closed a new deal and the salesperson or operations team then sets up a new project in Harvest to record our time and expenses against, and something we want to keep a close eye on is situations where a salesperson has closed the deal but done so by agreeing a daily rate below the rate we’d normally charge.

To keep an eye on this type of deal being agreed and to make sure the salesperson involved knows that the discount will be coming out of their commission, we’ve setup another Zapier workflow that detects new Harvest projects being created, runs a look within Looker that returns the average rate charged across all projects over the past three months and sends a message to the salesperson concerned if the deal rate is below this average.

revenuebot.png

In the future we can change how this average rate is calculated, introduce other data and analytics into the calculation and by using Looker we’re making sure these analytic insights are governed and delivered “as-a-service” to the requesting business process. If you’re looking to use Looker Data Platform to integrate, inform and help orchestrate your business processes then drop us an email and we’d be happy to share our experiences so far.

New Features in Looker 6.16 : Conditional Alerts (Beta), Content Curation (Beta) and LookML IDE Folders

Looker release 6.16 is rolling-out to customers right now and comes with three new features that you might find interesting:

  • Conditional Alerts (beta),

  • Content Creation (beta), and

  • Folders with the LookML IDE

Note that to enable beta features such as conditional alerts and content creation boards in Looker you’ll need to log in as an admin to specifically enable them before they’re available for use, and as experimental features they are not fully developed and may significantly change or be completely removed in future releases.

Conditional Alerts (Beta)

You can create a form of “alert” in Looker right now by scheduling a look to be sent to you only when a report returns rows, doesn’t return rows or returns a different set of rows than last time, but this is a pretty obscure feature that most end-users won’t have heard of. The new Conditional Alerts (beta) feature in Looker 6.16 makes creating alerts much simpler and is enabled from the Labs section of the Admin page in Looker.

Untitled 3.png

Once you’ve enabled this beta feature, you’ll then see “alert” icons in the top right-hand corner of every dashboard item, like this:

Untitled 4.png

Click on the bell icon to define the alert. In the example below I’ve configured the alert to send me an email if net margin has dropped below 20% at the end of the previous day, with the email going out at 9am next day to give me a heads-up that action needs to be taken - but also saving me the time of scanning the dashboard each day just to the KPI is within target.

Untitled 5.png

Alerts created by users are now listed alongside scheduled looks on the Looker Admin page together with a history of previous alert executions.

Untitled 6.png

Content Curation (Beta)

Boards are a new type of content object within Looker 6.16 and give you the ability to “pin” dashboards and looks relating to a particular topic or department together in a single place. Like conditional alerts, the new Content Creation / Boards (Beta) feature has to be enabled by an admin before it becomes available to end-users, but once you’ve done so then a new Boards (beta) menu item appears in the Browse page menu.

Untitled 8.png

Click on the Pin a Dashboard or Look button to bring up a dialog listing all of the dashboards and looks in spaces you’re able to access, like this:

Untitled 9.png

You can then add more dashboards or looks, and create sections within each board to further organize your content, as I’ve done here with our internal analytics board.

Untitled 7.png

IDE Folders

The final new feature I wanted to highlight isn’t a beta, but it does need to be enabled for each individual LookML project before use. To switch on the new folders feature for your LookML project first select Project Settings from the drop-down menu in the LookML developer IDE:

Untitled 3a.png

Then, check the Enable Folders checkbox just above the Save Project Settings button.

Untitled 3b.png

Now, press the “+” button to see a new Create Folder option.

Untitled 3d.png

Then once you’ve given your folder a name, you can move existing view, model and other LookML object files into the folder you’ve just created. Note how the various LookML files have different icons now for views, models etc.

Untitled 3e.png

Release notes for Looker 6.16 are on the Looker website, and contact us now if you’d like to find out how we can help you get moving with your Looker implementation.

Rittman Analytics is now a UK Consulting Partner for dbt ("Data Build Tool")

We’re very pleased to announce that Rittman Analytics is now an official Consulting Partner for dbt, working with our clients, the community and the wider dbt ecosystem to get the most out of this open-source analytics framework and accompanying commercial dbtCloud service.

You can read more about the role of Consulting Partners on the new Ecosystem page on the getdbt website and while you’re there, check-out comments from real-world users of dbt including one of ours, an excerpt from our recent blog post on How Rittman Analytics does Analytics.

In that blog post we talked about how Rittman Analytics used dbt as part of our modular “extract, transform and load” data loading approach when preparing data ready for use with Looker:

ra_analytics_architecture.png

Since launching the company last year we’ve built-up a fair bit of experience implementing dbt and dbtCloud on several clients projects in areas such as

  • transforming event-based (Segment, mParticle), data pipeline (Stitch, Fivetran) and batch-loaded raw data sourced from SaaS, Telco, web application and other data sources

  • using Snowflake Data Warehouse, Postgres and Google BigQuery as sources/targets

  • integrating with GitHub, AWS CodeCommit for local/remote branch-based git development

  • creation of automated CI/CD dbt build test pipelines using dbtCloud and AWS CodeBuild/CodePipeline

If you’re new to dbt or interested in how we use dbt and dbtCloud both internally and on client projects, check out our three recent blog posts on this topic:

and conversations we’ve had in the past with Tristan Handy, CEO and co-founder of Fishtown Analytics (primary sponsors of dbt) on our Drill to Detail Podcast:

Contact us now or email info@rittmananalytics.com if you’d like any help, or advice on your dbt implementation - we’re UK-based with clients in London, northern Europe the USA.

News on the Second London Looker Developer Meetup, 10th July 2019 at GoCardless, London

The second London Looker Developer Meetup is less than two weeks away now, running from 6pm - 9pm on Wednesday July 10th 2019 at the GoCardless offices in Goswell Road, London. Co-hosted by Mark Rittman and Jon Palmer from GoCardless, join us for an evening of chat, networking, presentations and Q&As around the Looker platform and the following agenda:

  • 6:00 - 6:30pm: Registration, Food & Networking

  • 6:30 - 6:40pm: Welcome and Introduction - Jon Palmer, Head of BI at GoCardless and Mark Rittman, CEO of Rittman Analytics

  • 6:40 - 7:00pm: “Getting Started with Looker” - Baha Sahin, Onfido

  • 7:00 - 7:20pm: “Balancing Liquidity in an On-Demand Staffing Marketplace” - Charles Armitage, Florence

  • 7:20 - 7:35pm: Update from Looker - Dave Hughes and Zara Wells, Customer Success at Looker

  • 7:35 - 8:00pm: Data Analytics Panel

  • 8:00 - 9:00pm: Networking and close

We’re particularly looking-forward to Charles Armitage’s presentation as Florence were Rittman Analytics’ first ever UK client when we launched the company back last year, and we’ve continued to work with Charles and the team since our original engagement right through to their recent successful Series A funding round.

The First London Looker meetup ended-up oversubscribed so make sure you register now if you’re planning on attending, and I’ll be joined by the Rittman Analytics team on the night if you’d like to chat with one of us about your Looker implementation - drop me an email at mark.rittman@rittmananalytics.com if you’d like to schedule a chat in-advance.

Rittman Analytics is now a Segment Certified Implementation Partner

In our recent blog posts on How Rittman Analytics does Analytics: Modern BI Stack Operational Analytics and Customer Journey and Lifetime Value Analytics we talked about using services from our partners at Stitch, Fivetran and Segment to connect and integrate data ready for analysis with Looker.

So far we’ve worked with joint Segment/Looker/Rittman Analytics customers such as Let’s Do This to build customer and product analytics platforms on-top of Segment’s rich, event-level behavioural datasets. Now we’re pleased to announce that Rittman Analytics is now a Certified Implementation Partner for Segment, giving us the ability implement Segment Connections along with Segment Personas (Customer Data Platform) and Segment Protocols (Data Quality) on client analytics and wider digital marketing / digital transformation projects.

segpartner.png

Look-out for more information on how we’re using Segment technology together with modern, modular cloud analytics services from Looker, Stitch, Fivetran, Snowflake and Google Cloud Platform over the coming weeks, and if we can help with any of your Segment implementation questions just get in-touch at info@rittmananalytics.com.

Continuous Integration and Automated Build Testing with dbtCloud

If you read my recent blog post on how Rittman Analytics built our operational analytics platform running on Looker, Stitch, Google BigQuery and dbt, you’ll know that we built-out the underlying data infrastructure using a modular Extract, Transform and Load (ELT) design pattern, like this:

ra_analytics_architecture (1).png

We version-control our dbt development environment using git and Github.com, and do all new development beyond simple bug fixes as git feature branches, giving us a development process for dbt that looked like this:

original2.png

1. We’d start by cloning dbt git repo master branch from github.com to the developer’s workstation, which also would have dbt installed locally along with the Google Cloud SDK so that they can connect to our development BigQuery dataset.

clone.png

2. Create a new, local git branch for the new feature using the git CLI or a tool such as Github desktop

newnrahc.png

3. Develop the new feature locally using the developer’s install of dbt, committing any changes to the feature branch in that developer’s local git repo after checking that all dbt tests have run successfully.

locals-imac:ra_dw markrittman$ dbt run --models harvest_time_entries harvest_invoices --target dev
Running with dbt=0.13.1
Found 50 models, 8 tests, 0 archives, 0 analyses, 109 macros, 0 operations, 0 seed files, 35 sources

20:53:11 | Concurrency: 1 threads (target='dev')
20:53:11 | 
20:53:11 | 1 of 2 START table model ra_data_warehouse_dbt_dev.harvest_invoices.. [RUN]
20:53:13 | 1 of 2 OK created table model ra_data_warehouse_dbt_dev.harvest_invoices [OK in 1.80s]
20:53:13 | 2 of 2 START table model ra_data_warehouse_dbt_dev.harvest_time_entries [RUN]
20:53:15 | 2 of 2 OK created table model ra_data_warehouse_dbt_dev.harvest_time_entries [OK in 1.70s]
20:53:15 | 
20:53:15 | Finished running 2 table models in 5.26s.

Completed successfully

Done. PASS=2 ERROR=0 SKIP=0 TOTAL=2
locals-imac:ra_dw markrittman$ dbt test --models harvest_time_entries harvest_invoices --target dev
Running with dbt=0.13.1
Found 50 models, 8 tests, 0 archives, 0 analyses, 109 macros, 0 operations, 0 seed files, 35 sources

20:53:37 | Concurrency: 1 threads (target='dev')
20:53:37 | 
20:53:37 | 1 of 4 START test not_null_harvest_invoices_id....................... [RUN]
20:53:39 | 1 of 4 PASS not_null_harvest_invoices_id............................. [PASS in 1.50s]
20:53:39 | 2 of 4 START test not_null_harvest_time_entries_id................... [RUN]
20:53:40 | 2 of 4 PASS not_null_harvest_time_entries_id......................... [PASS in 0.89s]
20:53:40 | 3 of 4 START test unique_harvest_invoices_id......................... [RUN]
20:53:41 | 3 of 4 PASS unique_harvest_invoices_id............................... [PASS in 1.08s]
20:53:41 | 4 of 4 START test unique_harvest_time_entries_id..................... [RUN]
20:53:42 | 4 of 4 PASS unique_harvest_time_entries_id........................... [PASS in 0.83s]
20:53:42 | 
20:53:42 | Finished running 4 tests in 5.27s.

Completed successfully

4. All dbt transformations at this stage are being deployed to our development BigQuery dataset.

5. When the feature was then ready for deployment to our production BigQuery dataset, the developer would then push the changes in their local branch to the remote git repo, creating that branch if it didn’t already exist.

commit.png

6. Then they’d create a pull request using Github Desktop and the Github web UI summarising the changes and new features added by the development branch.

create_pr.png

7. I’d then review the PR, try and work out if the changes were safe to merge into the master branch and then accept the pull reques, and then overnight our dbtCloud service would clone that development git repo master branch and attempt to deploy the new set of transformations to our production BigQuery dataset.

And sometimes, if for whatever reason that feature branch hadn’t been properly build-tested, that scheduled overnight deployment would then fail.

fail.png

So having noticed the new Build on Pull Request feature that comes with paid versions of dbtCloud, we’ve upgraded from the free version to the $100/month basic paid version and added an automated “continuous integration” build test to our feature branch development process so that it now looks like this:

DIAG2.png

To set this automated build test feature up, we first linked our Github account to dbtCloud and then created a new dbtCloud job that triggers when a user submits a pull request.

buildtest.png

8. Now, when I come to review the pull request for this feature branch, there’s an automated test added by dbtCloud that checks to see whether this new version of my dbt package deploys without errors.

buildstep.png

9. Looking at the details behind this automated build test, I can see that dbtCloud has created a dataset and runs through a full package deployment and test cycle, avoiding any risk of breaking our production dbt environment should that test deployment fail.

pass.png

10. Checking back at the status of the test build in Github, I can see that this test completed with no issues, and I’m therefore safe to merge the changes in this feature branch into our development master branch, ready for dbtCloud to pick-up these changes overnight and deploy them as scheduled into production.

testpass.png

Docs on this recent dbtCloud feature are online here, and you’ll need at least the basic $100/month paid version of dbtCloud for this feature to become available in the web UI. It’s also dependent on Github as the git hosting service, so not available as an option if you’re using Gitlab, BitBucket or CodeCommit.