Ben Gilbert: Tracking Network Effects

Posted on Wed, Jan 18, 2017
 

This article is part of a series called “The Data-Driven Product Manager,” interviews with product experts to help you use data to improve your product.

I met Ben Gilbert at the Industry Conference in Cleveland in September, where he presented a talk called “Universal Truths in Birthing a New Product — at a Pre-seed Startup or the Biggest Software Corporation in the World.” He hypothesized that companies with the ability to scale fast (and therefore be attractive to investors) usually incorporate “network effects.” These are ways that growth tends to lead to exponentially more growth, because the value of the product for the user depends, at least in part, on more people using the product. Great recent examples would be Airbnb, Slack, Uber and Amazon, to name a few.

From Ben’s talk at Industry: The Product Conference.

Ben is a founder and partner at Seattle’s Pioneer Square Labs, whose unique approach develops business ideas and pairs them with the right entrepreneurs to bring them to market. Building on the Lean Startup principles of iterate and fail fast, PSL tests if ideas are marketable, and then rolls out companies that have a head start in the market. To do this, they rely on both instinct and data to move forward and measure success.

Before joining PSL, Ben co-founded Madrona Venture Labs, and ran The Garage, Microsoft’s Grassroots Innovation program for side projects. Ben has a background in iOS and web development, with past stints at Microsoft, ExactTarget and Cisco. Along with a passion for innovation and entrepreneurship, Ben’s an Eagle Scout who loves outdoor adventure and is also the host of the Acquired podcast.

We had a great conversation and I’m excited to share some of Ben’s thoughts with you here. 

Data tracking should start immediately

I first spoke to Ben about how the companies he works with at Pioneer Square use data to develop their product and their market fit.

In all the companies that we start, we’re absolutely vigilant about full data instrumentation from the beginning. Before, there’d be this weird three-month gap around data at the beginning of the company, so it was like, “Well, we have analytics on,” but it wasn’t like we really thought it through. For example, a company would say, “Yeah, we had Google Analytics set up,” but now it’s like, “Did we actually put events in, did we properly add UTM parameters to everything so we understood when people are coming from this source, and they had this kind of lifetime customer value?”
We’re actually using this company, right now, called Segment that’s helped a lot. It’s unlike every other analytics platform I’ve ever used that has some non-ground truth version of your data that they store and you don’t ever get to directly query it. With Segment, it actually lives in my production database. That enables massive developer power. It can help inform people across the organization that aren’t going to be writing code. It acts as the broker to all the other services and keeps all the historical data, so if later on you’re like, “I wish I had been using KISSmetrics the whole time,” you could just replay all your data into KISS, or into Mixpanel, or into Amplitude, or any analytics tool.

Cohorts and Correlations

Next, Ben and I discussed what metrics were important to the very early stages of a startup and how that impacts a company’s ability to scale later on.

Now what we try to do in all the companies that we start is attribute as much as we possibly can to the marketing source. Whatever we can possibly learn about when a user came in, how we got them, really looking at cohorts, not only by time, but understanding what groups of people, later on, become very valuable customers.
LTV vs. CAC (image from  Notion)
A lot of start-ups get to this point of assessing “cost of customer acquisition” and they don’t consider lifetime customer value, the other half of the equation. It’s really hard to do that at first. You don’t have enough data to consider the lifetime customer value, so you just default to “how cheaply can we acquire people?

"A lot of start-ups get to this point of assessing “cost of customer acquisition” and they don’t consider lifetime customer value, the other half of the equation."

You’re kind of screwing yourself over later if you didn’t track who your customers are, and how do some customers perform versus others a year later, looking at transaction history. This tells us how we acquired the most valuable customers, and if we can spend twice as much or ten times as much acquiring them if they’re going to be a power spender on our platform.

Data for everyone

We also discussed how important it is that everyone on the team can access data, and how non-engineers can understand the data that’s being tracked.

We have two worlds with data. Analytics from your production data for engineers, and analytics from your tools that you don’t need engineering to access. We’re seeing an interesting trend now where those Venn diagrams are merging so teams have the ability to do all the dirty work, but can also use another tool or build their own tool on top for everyone to use.
I’m a developer also, and I’d get asked a question by someone at a meeting when we’re looking at data, deciding if a company is viable and we should spin out. Like when we’re looking at the Mixpanel dashboard, and I’d be like, “Uh, I don’t think that’s tracked in analytics. I mean, I guess we have it in all the timestamps from user signups and actions they took in our app. I can try and figure that out from our database.” I’d be doing that, and then I’d get another question that was available in the analytics.
Sharing data with everyone on the team via  Notion.
It seemed like these worlds were stupidly separate, and I was frustrated with that. I’m still new to this newer stack with our Segment implementation, but I think tools that allow everyone access are the future, where you get to answer all the questions from one place instead of bouncing back and forth between two, requiring two completely different skill sets.

"I think tools that allow everyone access are the future, where you get to answer all the questions from one place instead of bouncing back and forth between two, requiring two completely different skill sets."

Teams tend to divide along data, with marketing always tracking acquisition and product always tracking engagement. I think the success of companies comes in marrying those two things together and understanding that your acquisition doesn’t matter unless there’s engagement. What really matters is how long you can extend that user lifetime.

What we know we don’t know

I asked Ben what kind of pitfalls he’s encountered on his road to being more data-driven.

In general, I’ve become much less trusting on any sort of acquisition data, because things change. I’ve over-trusted and I’ve over-represented the cost to acquire a customer without thinking about how deep the well is.
One thing that I definitely think about that I didn’t used to think about is “what are the unique conditions right now that I’m not factoring into my tests that are affecting all the numbers that are coming out of it?” A lot of that is like seasonality or things I don’t yet know about the business. Whenever you’re starting a new company, there’s so much about the space that you don’t necessarily know that could be impacting it.

“What are the unique conditions right now that I’m not factoring into my tests that are affecting all the numbers that are coming out of it?”

We’re working on a business right now called LumaTax that actually just spun out. LumaTax has an extremely seasonal period, because it’s small business sales tax filing. Nobody cares about sales tax until they get the notice from the government that they have to file their quarterly sales tax, so you get these really weird burst volumes where you actually should have all your marketing spend in one month. If you’re prototyping during a tax filing month versus any of the other ones, it’s completely different. And I didn’t know before if it was quarterly or monthly filings, so I couldn’t have even predicted that there would be a heavy month.

Customer Data FTW!

Ben shared a great example of a company that’s used data to their great advantage by tracking key details about their customer value.

I heard a really good anecdote about Zuily in the early days. They had a range of customers that they knew they could acquire for 50x more than other customers, because they were so good at understanding what the later behavior of those customers would be. They literally were spending 50 times more on some customers because they had confidence it was worth it!
I think the vast majority of start-ups would not know that. They would have one or two numbers, and they’d go, “Well, for this type of customer, we’d pay this; for that type of customer, we’d pay that.”

"The vast majority of start-ups would not know that. They would have one or two numbers, and they’d go, “Well, for this type of customer, we’d pay this; for that type of customer, we’d pay that.”

I think what I’m aiming to do in all the companies that we start is have the ability to really slice and dice as much as we can know about a person so that we can say, “Yeah, it’s worth paying 100X the acquisition cost for this customer compared to that customer.”

Thanks again to Ben for taking the time to talk with us. Check out some other recent posts in the series here and stay tuned for some other great conversations from Industry Conf. You can also learn much more about using data for product development in our School of Little Data, a free email course we created to help you get started.

Read...

New in Notion: Multi-project support for Pivotal Tracker

Posted on Tue, Dec 20, 2016
 

Pivotal Tracker is a project-planning tool that enables development teams to manage projects with a straightforward, agile approach. Notion helps teams organize their team performance data alongside their app performance data in order to make faster, smarter decisions as a team.

With our latest release, we’ve made it even easier for teams running multiple projects to collaborate on their data with Pivotal Tracker dashboards.

Read...

Test Big Strategies via Small Experiments

Posted on Mon, Dec 12, 2016

This article is a guest post from Nichole Elizabeth DeMeré, SaaS Consultant & Customer Success Evangelist.

“The best way to test a big idea or strategy is through small, fast, additive experiments.” – Ash Maurya, Scaling Lean Tweet this quote.

Lean startup methodology is predicated on the idea of making a minimal investment (and taking minimal risk) to learn the most valuable lessons. When it comes to creating highly effective strategies, the method is the same: creating small, fast, low-risk, highly informative tests.

Read...

Notion for Github

Posted on Tue, Dec 6, 2016

Developers already know about Github. The collaboration tool has over 10,000,000 customers in 60,000+ organizations, making it one of the most widely used tools used by product teams. But not everyone on a product team knows how to use it, or even has an account, which can mean it's difficult for managers and stakeholders to get full visibility into Github metrics.

Our latest integration, Notion for Github, solves this problem with easy-to-build Github metric dashboards.

Read...

Notion for Intercom

Posted on Mon, Dec 5, 2016

Intercom is one of the more widely used customer communication platforms, servicing over 13,500 active installs with a series of products designed to increase customer engagement.

Read...

Build, Measure, Learn, + Communicate

Posted on Mon, Nov 28, 2016

This article is a guest post from Nichole Elizabeth DeMeré, SaaS Consultant & Customer Success Evangelist.

Build, measure, learn. They’re the fundamental building blocks of Lean Startup methodology. Critics equate this 3-part cycle with tossing half-baked products to consumers to see if they work – a mistake easily made if you don’t fill in the spaces between Build – Measure – Learn with a little common sense and a lot of communication.

Read...

How to Create a Lean Startup Culture

Posted on Mon, Nov 14, 2016

This article is a guest post from Nichole Elizabeth DeMeré, SaaS Consultant & Customer Success Evangelist

Some describe work ‘culture’ as a “learned process,” or “a set of rules and standards,” or even behaviors that fall within a proper and acceptable range according to the organization. Those sterile definitions may work for traditionally corporate environments, but startups? No. Startups have a culture all their own.

 
Read...

New in Notion: Mix and match and measure with more integrations

Posted on Tue, Nov 1, 2016

SaaS tools are a staple of any company’s infrastructure. From growing startups to teams within large organizations, companies rely on dozens of tools to get their jobs done and are constantly looking for more data from these tools to improve their process. That’s why we’re excited to officially announce our next batch of integrations with Bitbucket, Github, Intercom, Mailchimp, Mailgun, and Mixpanel.

Read...

5 Reasons you're failing at Lean Startup Methodology

Posted on Tue, Nov 1, 2016

This article is a guest post from Nichole Elizabeth DeMeré, SaaS Consultant & Customer Success Evangelist.

Most startups fail – that’s the premise of The Lean Startup, its methodology and analytics. It’s all about setting up your startup for success by waiting to build until you know exactly what your customers need and will pay for, and then embarking on a journey of data-driven improvements.

Read...

A Product Manager Communication Survival Guide (or how to tame information overload)

Posted on Wed, Oct 12, 2016

This article is a guest post from Nichole Elizabeth DeMeré, SaaS Consultant & Customer Success Evangelist.

It’s all you, baby.

Or, more accurately, it’s all on you.

The burden of communicating among teams, in between departments, and being the go-to get-it-done-guy/gal for CEOs and managers – it all tends to fall heavily on the Product Manager’s shoulders.

Product Managers are the linchpins of their organizations. The fillers of “the white space” - the processes and tasks that need to happen, but for which no one is specifically responsible.

Read...

Track all of your KPIs with ease in one place.

Use Notion with your team, free for 30 days.

Get started free.