In the third episode of MOOPs TV Justin Norris shares how he made a mistake on a client account, but the interesting twist is that it actually took a while to discover the error!
Watch the episode to find out what the mistake was and how (if possible) it was resolved.
Avoid your next MOOPs with Automated monitoring across your entire marketing & sales tech stack. Learn more.
Episode 3 – Justin Norris
Mike Rizzo: Justin, thank you for coming on to record, what we found they referred to as MOOPs TV. So sharing your marketing ops mistakes, as with all of our guests, we deeply appreciate you taking the time to, to share and have an opportunity for us all to learn from. so why don’t you kick us off and some introductions.
Tell us a little bit about who you are and,what you’re currently working on or doing, in your role and just a little bit about your background and years of experience in marketing operations.
Justin Norris: Sure. Thanks, Mike. Appreciate you having me. It’s good to be here. And I’m Justin Norris.
I’m the senior director of solutions architecture at Perkuto. We are a MarTech Marquetto and we’re Cado focused consultancy. And I’ve been working in. Let’s say marketing more broadly. It will be coming up on 15 years now, but for the past 10 years really focused on the rev ops and marketing opposite kind of business systems space, first in house.
Justin Norris: And then for the last six years in consulting where, you get a chance to learn a lot of things. So you get a chance to make a lot of mistakes too. So I’ve done my share of both.
Mike Rizzo: Yeah. Yeah, absolutely. It’s quite a different world. Being in-house versus, in the consulting side of things, it’s lots more to interact with and touch and I’d probably opportunities to break things.
Justin Norris: Yes. Yep. There’s a, hopefully you don’t break too much, at least not in an irreversible way, but yes. There’s no shortage of either.
Mike Rizzo: Yeah. Good stuff. so at what point in your career did this mistake we’re about to hear about really take place. Was it early, mid. Yesterday.
Justin Norris: There’s a, as I say, there’s no shortage.
and I, when folks get started in consulting, there’s often a little bit of concern I find around, oh no, am I gonna, there’s a hesitancy, am I going to make a mistake? And, we try to create a safe space, for making mistakes. We’ll put guardrails in place to limit the impact of mistakes.
but also. have everyone understand that it’s going to happen. And I often say I’ve made bigger and worse and more expensive mistakes than anyone. So if I can do it and live to tell the tale and I’ve been around for a while, then, you’ll be okay. But this mistake, it was midway through my time for Crudo.
Justin Norris: It actually happened probably around 2018. Although the impact of it was not detected, for some time. And then the fix for it took even longer. Oh,
Mike Rizzo: interesting. This is juicy. I’m excited. Okay. Wow. That mistake that actually took a little while. This is good. Good.
Justin Norris: So the mistake was actually on the surface of it quite trivial. It was a Marketo and Salesforce integration. And, for anyone who knows a little bit about the Marketo Salesforce integration, I know not everybody does, but. When you’re integrating those two systems, you, define what fields you want to include.
you define permissions on those fields, whether they’re read only or writeable. and then there’s a final step that you can do in Marketo, which is called field blocking rules. And what those rules do is to find, which systems or actions are. Overwrite an existing value in a field, which is a useful feature.
Justin Norris: So in Marketo you can say, all right, Marquetto can write to the job title field for a new lead for the address field for a new lead. But if there’s already a value there, I don’t want, a form fill to be able to overwrite it or a form field could override it, but a list upload can’t so you can actually get quite granular and decide which source.
can overwrite. And sometimes this is not something that people that interested in, or maybe they’ll do field blocking for a few things, but it’s not a big concern, but in this particular issue,it was a complex environment, very large company, and, And a complex project for that reason with a very complicated as well and well-developed and established Salesforce organization.
Justin Norris: And there was a particular concern from this, client, particularly their Salesforce team that Marketo is not going to overwrite Salesforce data. So we spent a lot of time actually establishing what those field blocking rules should be at. In this case, they were actually very. And, the irony of the whole thing is that it was in many respects, a model project up until that point, and we do a lot of testing on everything we do.
QA is a big part of our process and probably went through at least 30 different individual unit tests and QA, testing, just in Salesforce, the permissions of the sync, user testing. And Marquetto was, the systems are integrated. We tested it in a sandbox environment. we integrated those systems first and then we did it.
Production. And then the last and final step was to apply the, the field blocking rules and it just didn’t do it. there’s no other way to, to describe it. it just got missed. And specifically by me, I say we, but really I was the lead consultant on the project and was executing it.
So it was, I forgot. And. we S we celebrated the successful launch, the integration, and it’d been a long road to get there, with a lot of change management, different things. and, winning over the Salesforce scene, there’s a variety of aspects of the project beyond just your standard integration.
So we launched for successful. And then I don’t know exactly when it was, but it was some, maybe eight to 10 months later, perhaps that, at first came to light that Salesforce data was being overlooked. Five Marquetto.
Mike Rizzo: And how do I, who noticed
Justin Norris: that? I think it was probably the Salesforce team.
Yeah. That noticed it, which is, this is a stakeholder that we’d spent a lot of time building up trust with. And so you’re definitely not the team that you wanted to have noticed that. but yeah, as I recall, it’s a bit fuzzy for memory, but somebody brought it to our attention, maybe gave a report and said, Hey, what are these values?
How is this possible? you start digging. And, and I pretty quickly identified, oh no, we didn’t set the field block and we did still feel bloggers. So what could we do? obviously I had to take accountability for it and just explains it here it is. We didn’t do it. It was a mistake.
It was a miss. we’re going to fix. And so the process for fixing it was a number one, trying to identify the records that were impacted. And we were able to do that, through some smart lists and Marketo and some reports over on the Salesforce side and the types of data being overwritten, in some cases it might just be, they had a nice clean data for a contact, and then that person filled out a form of Marketo and maybe they.
capitalizing their information in the way that, we would have flagged or in some cases they’re submitting other data, which was considered less trustworthy, the results of some list, stock loads, which might’ve been impacted. So there was a soup of changes that came from. About how
Mike Rizzo: long did that go on now?
are we saying like a year,
Justin Norris: like four weeks before we noticed it or before? I can’t remember exactly at what point we noticed it. I was just looking back through some history to see how long it was till we actually fixed it. And it was about a year later when the final fix describe when life. So there was a big history of data.
And it was fortunate, a big history of data that had gotten changed. It was fortunate that we noticed when we did or that at least that the Salesforce team noticed when they did, because, Marquetto has a,a window of data retention when the changes are stored. and if it had gone beyond.
Some of those previous values could have been not irretrievable or at least certainly not as easily retrievable. So it was good that we found out when, the history of changes were still extractable. So basically what we had to do is identify the records that were affected and then extract from our kettle, the, using the API, the history of a data chain.
And fortunately a Marketo, when a data I’m going to change to a field is logged, it’ll log the, the new value and it’ll log the previous value. And those are both stored as attributes of an activity. And you can, through that, take a particular change and essentially reverse it by saying, this is what it was before.
And I’d like to go back to that. Now, of course, things are not. so straight forward, because in some cases there would have been a lineage of multiple changes. various stages of writing, maybe Salesforce was then subsequently updated again. And so we had to go through a whole process of defining what are the business rules, for this change, like, all right.
If. Marquetto was the most recent update. then revert back to the most recent Salesforce update, but not the most recent Marquetto update, but if Salesforce was the most recent update, then just keep that value. Don’t do anything. There was also exceptions, for example, the unsubscribed field, we would allow Marquetto to have priority because that was considered right.
Higher source of truth. but only if it was. true. But if it was, You start
Mike Rizzo: that to these extended as weeds.
Justin Norris: But we had to, and, so the documentation just of the fix for this could have been, 10 or 20 pages long. And then we essentially had to build an application.
That would extract all these changes from the API and then process them and, produce a canonical value. So it would say for this record, for this field, this is the canonical value. And then the testing on that was extensive. we had to value. the QA of that solution was probably more extensive than the original QA, or you can imagine, Because, cause it’s complicated, you have to go through each of those scenarios that I just described. And is it working? Is it not working? And in some cases it wasn’t working go back, refine the logic. and then you run into. trivial, but still important things like formatting issues or, non, standard characters that are being rendered and properly due to some kind of encoding issue.
And so each of those things, so it was some months of working on this solution with a few different developers, until finally. Ready and could then take that data, import it back to Salesforce, bring everything back to its, its original condition and then close the books on that one. but I tell you, I will never forget to do field blocking rules again, seared in my consciousness.
Mike Rizzo: Yeah, it’s the, the eternal checklist is like ever present in your brain. Now you will always think
Justin Norris: about that. And that’s what it was. we have a very detailed, grid of QA tests that we would run through and we just didn’t have a row for that item. And it was just implicit or assumed that you would do the field blocking rules if there’s just a row, which there is now that said.
Are the familiar are the field blocking rules complete based on the spec. then that would be fine. And that’s how it is today, but just the absence of that line. you’re reliant on the individual’s, awareness or memory of the need to do that in my case, it didn’t remember it in that instance and nine out of 10 cases, or even 90 out of a hundred K 99 at 100 cases, let’s say the impact of that mistake would have been limited.
Or none, but in this one particular scenario, it was huge because of the particular requirements of the client.
Wow. Yeah. hard lesson learned to make sure that the checklist has all of the right checklist things to check off, but Hey, yeah. This is the things that we go through and marketing operations, right?
Justin Norris: It highlights the importance of QA, generally, and no matter how smart you think you are, how experienced you are. I’d already, by that point, then consulting for, for four years, it’s not like I was a rookie in this game. and we have very good processes and we have extensive queue.
like in every area we were already going, meeting whatever the standard or beyond. The standard, that you would consider professional in that situation, but you can have gaps. And, another thing, another issue there was that I was also performing my own QA, which is so actually there was another MOOCs, I would say.
we have a process that it’s always, a peer that does the QA,a second set of eyes on everything. And in this case, if I’m remembering correctly, I believe, the project, uh,we were in a hurry, we wanted to show value. And so I did the integration, the start, I can test this myself a bit, what’s the harm.
and how would I involve that second person? maybe they would have remembered. So there was another compounding process error. So it also shows. I like, I don’t like when we make these mistakes, but I like when the mistakes are a result of breaking process, only because it highlights how important that process actually is.
Justin Norris: Absolutely. That the process is working. So when you deviate from it things,
Mike Rizzo: Yeah. Yeah. I think so many organizations feel that process is a four-letter word, right? It’s a bad word. And in reality, in marketing operations, revenue, operations, really any operational role process is what creates scale and efficiency.
And when you follow a process, you won’t end up spending many multiples of months and time and budget. And having to fix something, that for 99% of other people may not have been,clients may not have been a problem. And in this particular case, it was a pretty substantial one.
Mike Rizzo: Sounds like the outcome was make sure the checklist is updated to reflect all of that ends that are necessary to go through on that QA checklist. And then in addition to that, like you, I also need a second set of eyes because. Every email that I ever send to our community is because I didn’t get a second set of eyes.
Anytime I send an email, it’s likely because I had no one read it. that’s it. And, yeah, it was funny when you say about process being a four letter word, and I can totally relate to that, impression. And I think the issue is not so much about process. I think it’s just that there’s a lot of bad process out there and that’s what gives it the bad name and,our co-founder at Perkuto.
Justin Norris: Our chief operating officer is a lean six Sigma black belt. And so the process is very much woven into the DNA of kind of how we do things even from a very early stage, but it was like less than 10 people when I joined and we still had a level of process maturity. It was way beyond what you would expect.
But what I learned from that, and what’s helped is that, I lean processes. The name suggests is a process that actually it makes things better. It makes things more efficient. It makes quality more predictable. It’s not there to get in your way. It’s actually there to relieve the burden of having to think something through every time and to ensure, just a predictable output.
When you see what. Process could do. And what goes into designing it, then you really fall in love with it, not as something that’s there to slow you down, but actually there to speed you up well, preventing, know, the sort of issue that happened in this
Mike Rizzo: case. Absolutely. No, I love that. I love the idea of,this lean process and that’s, that’s a great way to.
Just playback of the importance of process. I appreciate you sharing that with us. what would you say as, final thoughts, to someone who’s, maybe made a similar mistake or, is about to make a similar mistake, and just general advice for those in marketing and rev ops.
Justin Norris: I think in vice, when you make a mistake is the same in rev ops, as it is in life, generally, it’s, take ownership, take responsibility, make it right. And the cost of doing that is there’s always going to be less than the cost of not doing that. I think it’ll be painful in the short term, but, In the fullness of time, you’ll look back.
You’ll be glad you did it. And you’ll be able to talk about it on a podcast one day, it’ll all be fine, but it certainly hurts. and, kudos to, my, my peers, my team, my bosses at the time, nobody gave me a hard time about it. We all just buckled together and got it right.
And, Maybe if I did it again, I would get a hard time. It was always okay. To make a mistake once. that’s understandable if it’s repeated, then there’s maybe there’s something there that really needs to be looked at, but things happen where
Mike Rizzo: we’re human. We’re human. Yeah, absolutely. great, Justin, thank you for sharing your moves moment or at least one of them with us.
Justin Norris: I got more, you can bring me back. we’ll do like a whole
Mike Rizzo: series. It’ll be a whole series of just Justin moves moments and we’ll just keep going. But yeah. No. Thank you again. Appreciate you taking the time. I feel like everybody probably learned something today, so
Justin Norris: I appreciate it. Thanks a lot.