Episode Transcript
Kyle James (00:01.038)
Hey, welcome to the AI Chronicles podcast. I'm your host, Kyle James. Today we're going to be discussing how a marketing attribution and analytics company called Provalytics is using AI inside of their own business. And we'll share the exact steps that you can take in order to implement AI for yourself. Now, before I dive into that, listen closely. Are you looking to implement AI inside of your own company or just struggling to get your AI to stop hallucinating? Speak to GPT Trainer.
GPT trainer literally builds out and manages your AI agent for you, eliminating hallucinations for good. Go to gpt-trainer.com. I promise you it'll be the biggest time saving decision that you've made all year. Trying to set up AI on your own is like trying to build a house from scratch. Sure, you could do it, but the time and frustration it's going to take you to get it finished just isn't worth it. It's a thousand times faster and safer to hire professionals.
Scheduled consultation today. Once again, that's gpt-trainer.com. Today I have with me Jeff Greenfield, who is a serial entrepreneur and CEO of Provolytics, an AI powered cookie free attribution engine built to decode the true impact of channels like CTV and fuel smarter marketing decisions. Hey Jeff, welcome to the show my friend. How are you?
Jeff Greenfield (01:22.258)
You're doing great, Kyle. Thanks for having me here today.
Kyle James (01:24.77)
Yeah, super excited to have you on the show. So give us a little bit background here. Like what is Provolytics and like, did it come to be?
Jeff Greenfield (01:31.302)
Well, Provolytics answers that kind of old age question of, you know, there's this famous saying that marketers say half the money I spend in marketing is wasted. The only problem is I don't know which half. that's been around since before digital. It's from like the 50s and 60s. But I gotta tell you that things are a lot more complicated now because there's so many different places that you can put your dollars.
Kyle James (01:41.248)
Mm-hmm. Yeah.
Jeff Greenfield (01:57.05)
Things are not as trackable as they once were even five years ago So it's really kind of a black hole and it's very very complicated So Provolytics was designed to help marketers find that missing half Stop wasting the money on it and then tell you where to redeploy those dollars in order to get more bang for your buck more revenue for your dollar spent in market
Kyle James (02:20.524)
Yeah, absolutely. Marketing is definitely like, it's such a powerful tool. I think there are so, there are so many companies like, I've tried the marketing. It's, it's been hit or miss, right? Hey, I just worked on this part of it, but the other half, didn't really work out. But for Provalytics, like I know that you're using a little bit of on the AI side, but like, tell me like, why, why did you start using AI in the first place? And what types of challenges were you, were you trying to solve with it?
Jeff Greenfield (02:46.738)
Well, the biggest challenge that you have out there in the marketing world is kind of the signal loss that's gone on since cookies and the signal loss that's happened because everyone is looking through this lens of clicks, if you will. And that's been going on since like the beginning of the internet. But what's happening now is that everyone is using click-based media in order to determine what's working and what's not working. Let me paint you a scenario and then we'll kind of dig into it a little bit more.
Let's say I've come up with this great garden hose to sell. And so I run some ads on Metta and you happen to come along, Kyle, and you see that you're scrolling through your Facebook feed and you see the hose, you stop for a second, but you don't click on it. And then an hour or two later on Instagram and you see the cool video. Now you watch maybe 15 seconds of the video and you say to yourself, hey, you know, I'm going to get this. But then you get distracted. You go on and do things. And then that weekend you go and...
You start your hose to water your plants and there's a leak in it. You're like, my God, I got to get that hose. You take your phone out and you Google the name of the hose and you click and you buy. And so in my analytics, it's going to show me that Google brought you in as a customer, no mention of Meta. Now translate that into a thousand or 2000 sales at the end of the month. And now at the end of the month, I cut my spend on Meta and I spend more money on Google.
Kyle James (03:46.766)
Yeah.
Kyle James (03:59.522)
Hmm.
Jeff Greenfield (04:09.436)
But we all know that meta is what built the awareness, drove you through the funnel. It just happened to be that Google was the person who checked you out at the cash register. That was it. So what we're missing there is we're missing that signal of those impressions. about 10 years ago, we used to be able to capture all that by utilizing cookies. But now there's so many different apps and places that people go that we have these walled gardens that are there where you can't get that data in and out.
Kyle James (04:09.568)
Hmm.
Kyle James (04:19.662)
Hmm.
Jeff Greenfield (04:39.546)
And so essentially, if you think about like a customer journey, what I can see is I can see the clicks that come to my website, but I can't see the clicks that go to Amazon. So that's a hole. And I can't see the ads that you actually are interacting with or hovering over or the ones you even get exposed to. just imagine I know the purchase, I know there were some clicks and I know there was some other stuff, but it's a bunch of holes that are there. So that is the big problem to solve is
Kyle James (05:04.526)
Mm-hmm.
Jeff Greenfield (05:08.966)
How do I complete that journey on an individual basis and then an aggregate so that it can help inform my decisions? And what ends up happening in today's world is that most marketers believe that you invest dollars to buy clicks. And as a result, they optimize based upon clicks. But the reality is, that marketing works by investing dollars to buy eyeballs, which builds awareness. And when awareness is built enough,
Well, that's when people will walk into your store. And if your store is online, that's clicks and that leads to sales. So imagine being able to optimize and make decisions at the top of the funnel where people are getting exposed versus the bottom. Then you would have known to reinvest your dollars into Meta versus Google. And that's the holes that we ran into. So what we looked at is we looked at the tools that were at our disposal. And luckily we didn't have to go very far.
Kyle James (05:57.6)
Hmm.
Jeff Greenfield (06:07.056)
because before digital marketing was around, the only type of marketing that was there was TV, radio, print, and direct mail. And imagine back in those days, there were brands like BMW, Pepsi, all these furniture stores that would do this big mass marketing, if you will, not targeted. And yet they were able to determine that that led to more sales. And they used a statistical modeling technique called marketing mixed modeling.
Kyle James (06:15.926)
Right.
Jeff Greenfield (06:36.026)
And it actually measured the incremental impact that ads had on revenue. But it was slow. It was a huge project. It was only something that was done once a year. And it got to be so uncomfortable. Most brands would forgo it year after year. They wouldn't even do it. And you needed three years worth of data. So imagine like you're a new brand. You can't even get this done because there's so much data that's required. And so what we did is that we looked at that.
the other problem was is that it gave you output at a very broad level. So would tell you spend more on search, but wouldn't tell you what keyword or campaign or ad group. So for today's digital marketer, it wasn't useful at all. So we used that technique. We knew that that technique worked at its basis to be able to connect the dots, but we knew that we needed to go deeper into it. And oddly enough, we actually went even further back in time.
So we combined that older school technique with a technique called Bayesian. Bayesian is what's driving a lot of the AI today in machine learning. And Bayesian is actually a formula that was created by a preacher in the 1700s. Yeah. And the reason that it's used so often is because it actually works the same way that human beings think and solve problems, which is
Your best guess. If I asked you to cut down a tree, you'd never cut down a tree before. And I said, estimate how long you think it would take. You would give me a number. And then if I let you cut for another minute, I would ask you to revise your number. You would revise it based upon the new information. So it's used a lot in machine learning and AI because you can update it very, very rapidly. And then we also grabbed another technique out of the University of Chicago from the early 60s called seemingly unrelated regressions.
Kyle James (08:20.866)
Mm.
Jeff Greenfield (08:34.662)
And so we stitched together all of these techniques. And when we first did it, it was like a five-part process where we would process data, we would take a look at it and analyze it, and then we would make a decision if there were any changes that needed to be made and then take it to the next part. So when we first built this out about three and a half, four years ago, it was a laborious kind of human involved technique where it was a lot of start and stop, a lot of start and stop.
Then we trained AI to be able to go in to do kind of the steps that the human being was doing. We watched it for a couple of months, and now we're able to take data from our clients and run it completely through the entire process without human intervention. So we used AI to kind of, if you will, supervise the process and know what steps need to happen. We're not done with AI at this point because there's a lot more
that we can do with the large language models to kind of extend the usefulness of the data, if you will.
Kyle James (09:37.708)
Yeah. So walk me through, because mentioning a kind of like the older approach, and then you took on really applied the AI within, the probabilitics. Like, walk me through that a little bit more in detail, like step-by-step, like what exactly is the process look like when you implemented AI versus before when you didn't even have AI that was utilizing your systems?
Jeff Greenfield (09:58.576)
Well, in terms of for us as a company, it's significant because there's a significant time savings. The process, the data itself takes as long as it takes to process. There's no speeding up of that because there's a significant number of mathematical calculations that have to go on that you can't speed that up. You can't guess at that. The math has to get done. But what actually happens is there's this first part
that we kind of call middle out processing. We consider ourselves to be kind of in the middle of this older school marketing mix modeling technique and this newer what we would call multi-touch attribution. But so we live in that middle and we really liked Silicon Valley, the show. And so we kind of grabbed the name from there, but that middle out processing looks at the causality between all of the different processes. And it actually builds out, if you will,
a funnel and creates a system of equations. And so that is being supervised by the AI to make sure those system of equations are actually correct. And then it carries it over to kind of the next step. then it's that in that next step, it goes through a process of mathematical calculations and it supervises that. So one of the things that it does is it looks at each of the different marketing campaigns that are running. There's a concept called
ad stock, advertising ad stock. And ad stock means that you can imagine like if you ran a Super Bowl ad, your website would crash, it would go crazy and your sales would go up for that day. But the reality of what happens is that an ad that is that strong carries on for weeks or months. And you can actually mathematically calculate that. So that goes through that whole process. It goes through and calculates that.
Kyle James (11:35.735)
Mm-hmm.
Jeff Greenfield (11:54.256)
The AI is monitoring all this and it goes through a whole series of steps. And then at the end, it actually has to validate itself. So we want to make sure that our model is actually predicting accurately. Now it goes through a step called K-fold, which is a validation technique that's used in AI and machine learning. And what it does is if we're looking at, let's say a year's worth of clients' data, we're looking at the marketing data, very granular, and then we're looking at, let's say, sales.
And so what it will do is that it will train the model for about a month where it gives the model all of the marketing data and says, here's the sales for every day. And then it holds back 20 % of the days and tells the model it has to predict. And it supervises that entire process.
Kyle James (12:43.948)
Wait, so why 20 % of the, is it just to verify that if it's correct, I guess, or?
Jeff Greenfield (12:51.064)
Right, because what you want to do is it's holding back 20 % of the answers to then see and compare how well is it predicting to the actual. And then we do that five times. So it actually holds back 100 % of the data. In our world, in the marketing measurement world, typically you would hold back when you would do one of those marketing mix studies like they used to do in the old days, you would hold back the entire last month.
Kyle James (12:59.893)
and
Jeff Greenfield (13:20.474)
and see how well that predicted. But our users are most interested in what just happened. So if we hold that back, we don't want to hold that back. So we actually end up holding back the entire set. Also, what ends up happening in marketing data is that typically in the last month, there's new stuff because someone is always testing. So by holding back 100 % of the data, we're actually using as the training set, not just that first month, but every single day.
actually becomes a training set for every single other day, which is pretty cool. And then from that, we're able to get the predictive ability of.
Kyle James (13:53.71)
So, yeah.
Okay. So, so it's very interesting, like the holding back of the data, because it's almost like it's a verification process that the AI is being tested on to see if, Hey, will they actually make that correct answer? And since doing that within the AI, like what types of like results did you see that were maybe kind of eyeopening? And then maybe in translation to some of your clients that you've worked with, like what types of results are they seeing because of this? Like really essentially it's kind of like a breakthrough technology.
within Provoletics.
Jeff Greenfield (14:29.638)
Yeah, it is. It's pretty amazing because what we find is that sometimes the biggest limit is the quality of the data that goes in. So the program and the AI, when you give it data that is correct and makes sense, it is predicting better than 90 % of the time. So there's two variables that you look at when you're training that data and you hold back the data.
you're looking at the R square is what they call it, which is how well the model essentially is predicting. And a perfect R square is a one. If you ever see a one that tells you there's an error right away. And the way to look at this is that, okay, like if I had an algorithm that could predict the stock market and it was like a 0.8, so it was predicting 80 % of the time, you could bet on that, that would be good.
So our models, when they have correct and accurate data, they're like 0.9, 0.95 in terms of their accuracy, in terms of predicting. The other big variable that you look at is something called MAPE. MAPE is the error rate. So every model that you're going to build, especially with AI, there's going to be an error range in it. And what you want with that number is you want it to be below 20, like in the mid.
to high teens and ours are typically in the single digits. And I say, typically when we have good data, when we get bad data, what ends up happening is the model says, hey, there's something else that's going on here that I can't explain because our model is very specific. We're asking it to look at the relationship between marketing and sales or marketing and orders or marketing and website traffic.
But sometimes there's other factors that are actually going on. But in terms of clients, what we're able to do with this is that one of the things that has really hurt a lot of folks these days is understanding how marketing that they're doing to drive sales to their website is actually driving sales to Amazon as well. And then also how sales and marketing that they're doing on Amazon is actually driving sales back to their website.
Jeff Greenfield (16:49.284)
It's essentially this halo impact that's actually going on. so clients are now able to see through Provalytics how their TV ads, the impact that those TV ads are actually having on Amazon. Because one of the things that's happened in the last year is that if you just follow the numbers, the number of people buying on regular websites is starting to go down. More and more people are buying on Amazon. But imagine a brand who's spending a lot of money on television.
Kyle James (17:13.902)
Hmm.
Jeff Greenfield (17:18.03)
And if they're just looking at their website, it seems like their TV ads aren't working because it's a totally different team that's running Amazon and they're taking credit for all the sales on Amazon. But we're able to demonstrate and prove that halo effect that's actually going on. The other big thing as well is taking that example of meta where you don't see what's happening on meta. A lot of our clients, because they're focused in on like the Google engine,
and Google keeps telling them to spend more and more money based on that click down at the lower funnel, they get to a point where their business can't grow. They can still spend more money, but their cost to get a new customer keeps rising and rising. And with Provalytics, we're able to come in and show them how they can light up and launch new channels and how they can take the same amount of money that they're spending right now and squeeze 25 to 50 % more revenue out of it
Kyle James (17:59.736)
Hmm.
Jeff Greenfield (18:15.59)
by moving it more towards the upper part of the funnel to drive more awareness.
Kyle James (18:21.07)
So, so let me ask you this, cause like, I feel like the, of the biggest things for a lot of companies is like, they look at one piece of data, but like what you, the example you used earlier, like the, the, the, guess, Google ads or the secondary ads, but like the primary was the meta was the first exposure. And then the, then the click happened, like, and then they go, okay, well there's more clicks from this type of investment and the marketing side, but the previous one with meta, you're like, oh, well maybe we should tone back because we're only getting views there. Like since.
Since now, what it seems like, provoletics is taking like a holistic approach and going, okay, here are all the different avenues and here's where the data is saying on this, not just this part, but on the meta side, like you should still keep investing into it because what they don't want to do is say, okay, I'm going to stop investing in one thing and reinvest in another. And then that actually does more damage than good. But what you're saying is, provoletics is almost like it gives you that
that full view of where exactly all the data is coming from. it gives you a better, I guess, I guess you would say like a visual as to like where you should actually be spinning your dollar towards marketing.
Jeff Greenfield (19:29.778)
Well, you're actually right, Kyle. We actually essentially pan the camera back. You know the old saying, you can't see the forest for the trees. So we pan that camera back so you can get that bigger, broader view. But also, since this is a predictive engine, we can also go in and say to the engine, hey, next month I want to spend X amount of dollars. And I'm willing to spend
Kyle James (19:37.452)
Hmm.
Jeff Greenfield (19:58.194)
you know, I'm willing to increase my spend on any one item, let's say by 30 % and I'm willing to decrease my spend by 30%. So we give the AI a range. So I want to go out 30 days and let's say I want to spend $100,000 more next month and here's my range and it'll run through all the possible combinations. So it's looking both at that ad stock, it's looking at diminishing returns that you get from things and it will spit out for you a perfect
plan down to a very granular line item like campaign ad group keyword that says, this is exactly how much you should spend over the next 30 days, 60 days, or 90 days. So it used to be that marketers would have to take essentially, hey, here's how we did last month. And then from that, figure out the math to compile. Okay, this is what I should do this month. But they don't have to do that anymore. The machines can take care of that.
And that's where the AI comes in. It's like, it to the magic genie, put in all the data and let it spit out the perfect plan for you.
Kyle James (21:02.7)
Yeah, like it takes the guessing away, which I mean, obviously the guessing there's risk with guessing because then you could lose, but then you could win. obviously winning more is obviously more favorable in a lot of situations. If you know, if you have the data and the clarity, as you said, like the, the, the camera zooms back out so you can see, but then also zooms in when it needs to. So looking into this next year, 2025, 26.
Jeff Greenfield (21:04.965)
That's exactly right.
Jeff Greenfield (21:22.789)
Exactly.
Kyle James (21:28.204)
What are some of the like ProVoletix upcoming AI initiatives and like, do you see AI playing maybe the biggest role in your, in your operations next?
Jeff Greenfield (21:36.966)
Well, it's interesting. There's a lot of different directions that we can go in. One is that we've heard from customers, they want the ability to be able to talk to an assistant and ask specific questions. And sometimes the data set in a dashboard is too big to figure that out. This is one of the problems when you're dealing with this much data is that literally we could have thousands of pages of dashboards. And so you can always answer questions.
The question is, what is the question going to be? We can never anticipate it. And that's a pretty straightforward direction to go in because we can easily feed in the finished data set and then feed in to any type of a large language model a little bit more kind of direction and then come back with, and then anyone could be able to go in, any of our customers could go in and query an engine for that. But where I want to take it and where the hardest thing is, is that
With this data, there's a couple subsets of users. One user is the person who's pushing the dials. But the more important users are the ones at the C level, the ones at the top who are using this in a more strategic manner. And typically, the way I look at it is the user who's pushing the dials, they get the Excel report, and they have to prepare a PowerPoint for their boss, the VP of Marketing. So the VP of Marketing now has this five-page PowerPoint.
and then they have to put it all together onto a single slide for the CMO. And the CMO has to prepare a bullet point for the CEO who uses it at a board presentation. What I want to use AI for is how to better tell the story and tell that executive story. That's what's really important because that's something that being able to tell that story and tell it very simply is incredibly important.
And where I see the future of this is using graphics and images to be able to tell that kind of the dollar story. I think that that's going to be where we're going to be looking at. Because that's the biggest issue for marketers is how do I as a marketer have a conversation with finance? Because really what I want as a marketer is I want a bigger budget. And how do I tell my story to finance where I don't lose them in the detail?
Jeff Greenfield (24:03.715)
and so that we can both speak the same language.
Kyle James (24:06.144)
Yeah, it's almost like a trans like a bit of a translation where like, just say like even sales or C-suite or your marketing marketing might have a message, but taking that message and communicating it to a CEO or CFO, like, cause they have a certain thing in their mind of like, I need to understand this, but the marketing, might look at it with different sets of eyes. So it's almost like a translation. Wouldn't you say
Jeff Greenfield (24:26.898)
That's exactly what it is, Kyle. Yeah, that's a thousand percent right on because they see things from a different perspective. It's like they have a totally different viewpoint. And that's one of the issues in today's C-suite is how do get these people to see that they're looking at the same thing, but they're just looking at it from a different angle. And my plan is I think AI can help us translate that later.
Kyle James (24:47.79)
Mmm.
Kyle James (24:52.694)
Yeah, absolutely. And for, as we start wrapping up here, Jeff, I definitely appreciate you have a wealth of knowledge and I'm just like taking mental notes on everything you're saying here. I'm sure with everyone else listening in, they're probably saying the same thing. Where can they go to learn a little bit more about you and maybe even about Provolytics?
Jeff Greenfield (25:10.332)
We'll learn more about Provalytics. You can go to Provalytics.com or you can just go to GetProva.com. I'm also on LinkedIn. We've also got, I should mention as well, under our resource section of the website, we put together a kind of a measurement attribution certification course. It kind of goes over the past, the present, and the future. takes you about an hour and a half. There's a quiz afterwards. You pass it.
You get a good certificate for LinkedIn. But the best thing about it is for anyone who's listening, who's into marketing, this marketing world is constantly, constantly changing. And the best way to understand where it's headed next is to study the past. And that's why we put that together.
Kyle James (25:57.336)
that. Love that. Amazing. Thank you so much, Jeff. And it's great to have you on and thanks for everybody listening in today. Remember, if you're looking to implement AI into your business today, don't try and do it yourself. The time of stress that the AI could cause just isn't worth it. Schedule a call with GPT trainer and let them build out and manage your AI for you. Once again, that's gpt-trainer.com to schedule your consultation. Signing off for now. Thanks again, Jeff. Awesome conversation.
Hope you all have a wonderful rest of your day and looking forward to seeing you on the next episode of AI Chronicles.
