The ICE-T Framework: how to know which experiments matter (before you launch them)

Is lifecycle marketing just email? Not even close, if you ask Allison Bryant.

In her Unspam 2025 talk, Allison, a veteran lifecycle marketer, introduces the ICE-T Framework: a practical tool to help marketers prioritize experiments, size their impact, and make a clear case for lifecycle’s value. With wit, sharp metrics, and real-world examples (plus a nod to rapper Ice-T), Allison brings clarity to a space that’s often overwhelmed by testing overload and AI hype.

TL;DR:

• Lifecycle is not just support. It's a growth engine.
• The ICE-T Framework stands for Impact, Confidence, Ease, and Time to Learn.
• Not all tests are created equal. Size them before you run them.
• Early, small wins build trust and unlock resourcing.
• Lifecycle marketers are experimenters and growth catalysts.

If you’ve ever struggled to explain why your work matters, this one’s for you.

Allison Bryant: Thanks, Logan. Great introduction. I feel teed up to chat with y'all today. It's a joy and a treat to be here to talk about the ICE-T Framework, which is sadly unrelated to the celebrated American rapper and actor. Fun fact. He is on Cameo. A less fun fact: he is expensive on Cameo. So this is as close as we're gonna get to him today. 

This ICE-T Framework came out of a problem many of us share. With email marketing, we could try so many different things to achieve our program goals: everything from copy tests and imagery to frequency, cadence, and offers. Now that we can do this with AI, there's truly an infinite buffet of things you could test, and it’s tough to know where to start and what to do.

I developed the ICE-T Framework, with help from many others, to figure out which experiments are most likely to help me reach my goals before actually running the tests. So, in a way, we’re looking into the future a little, like using a crystal ball. The other thing this framework does for me, and hopefully for you too, is clarify the case for Lifecycle Marketing.

What can Lifecycle Marketing do for your company, especially for people who don’t know much about email marketing? It gives them context to understand. Like, “Oh hey, if we make these changes, here’s what could become possible for our company, our business, and our users.”

This is a photograph of my face; this is my face in real life. I want to give you a little context about who I am and why I think I have something to offer you today. I'm Allison. I've spent the last 11 years in email marketing in some form or another. For the first six years, I spent time in the nonprofit world. I was involved in email, social media, digital marketing and fundraising, specifically for the Audubon Society and Brown University. 

Then, I pivoted and worked for two growth-stage startups for the past five or six years. One was a meditation app called Happier, and the other one is where I am right now: GlossGenius. We are a business software, payments, and bookings for salons, spas, tattoo parlors, and dog groomers. For these latter two roles, I was the first lifecycle marketer hired.

So I had to help people at the company understand what lifecycle marketing is, why it matters, and why we should invest in it. I also had to get some quick, early, meaningful wins to prove, like, "Hey, this is a good investment. This is why you should care about lifecycle."

The ICE-T Framework helped me do that. So today, we’re gonna learn two things. First, we’re gonna learn how to estimate the impact of a test idea. That means asking, “Hey, I’ve got this idea for a test, something I could change. If it works the way I expect, what’s that going to mean for the business? What kind of incremental conversions are we gonna get from making this change?

The second thing we'll learn is how to rank your test ideas against each other when the rubber meets the road and you have to choose a test, execute it, and learn from it.

So, let's get into it. Let's talk about impact sizing. Again, I'm just saying, if the test succeeds, what kind of impact will that have on our North Star metric?

We're going to go through a couple of definitions. For your North Star metric, I'm sure all of you know what this is. It's the number your company is driving toward. It comes up in town halls. It shows up in your OKRs, probably tied into team or company-wide goals.

Based on your business, what is the most important number that tells you if your company is succeeding?

I currently work for a SaaS company, so subscriptions are one of our most important North Star metrics. If you work at a nonprofit or you're a fundraiser, maybe it's the total number of donations. If you're in e-commerce, it might be purchase revenue. Basically, it's the one key number that everybody cares about. But a North Star metric is not always the best metric to target when you're testing, because a lot goes into that number. For example, the number of subscriptions depends on things like churn and retention, and it can be really hard to move that big number everyone is focused on. So when you're testing, you're probably going to look more often at an up-funnel metric. Up-funnel metrics are the user actions that are correlated with your North Star metric, right?

Things like, you know, I work for an app company, so we might look at app downloads, app opens. And if people do those actions, they’re more likely to subscribe. You could also look at feature adoption, clicks, email opens, form submissions, sales conversations, or other touchpoints. Basically, anything where you think, "Hey, if this number goes up, then a number I care about should also go up."

But that also means that if you're running a test and targeting one of these up-funnel metrics, you're actually working with two hypotheses.

The first hypothesis is, "Will my test improve the up-funnel metric?" For example, "If I change the subject line, will more people open the email?" or "If I change the design, will more people click on it?" That’s a real hypothesis. And you could be wrong. I’m often wrong.

Then there's the second hypothesis: "If I improve that up-funnel metric, will I also see a positive change in our North Star metric?" So basically, "If I can impact this smaller number, does that actually cause a change in the bigger number, or are they just correlated?"

As you start testing different up-funnel metrics, you’ll learn. Some of these things are more closely connected to your North Star metric than others, and that will help you choose which metrics to focus on and make better decisions over time.

Now we’re getting into a formula. This formula tells us, if the test succeeds, if the idea was right, how many incremental conversions we’ll get on our North Star metric from this test. To get that number of incremental conversions, we’re multiplying five numbers. That’s it. So let’s talk about what those five numbers are.

First, your total audience. How many people could potentially see this message? How many are qualified for it? How many is it actually relevant to? Then, you want to look at what I’m calling the baseline inaction rate. That means, how many people are not already doing the thing you want them to do, right? Because those people, they’re good. They’re already doing what we want them to do. We’re happy with them. They’re not the ones we need to worry about. Everybody else, they’re our opportunity. They’re the people whose behavior we want to change.

Next, we're going to look at our messaging engagement rate. Just out of curiosity, how many of you here are only sending email? You're not sending SMS or push, or in-app notifications? Okay. So some of you do. That was a pretty even mix, if you were looking around. One of the key questions is: if I send someone a message, how likely are they to see it and engage with it?

That might change depending on whether you send a text versus an email, for example. But you want to ask yourself, “Okay, I have all these people whose behavior I want to change. How many of them can I reasonably expect to actually pay attention and hear what I’m asking them to do?” Then you're going to look at your incremental action rate. That means, from the audience of people who haven’t yet done what we want and who are actually paying attention, how many of them do we think we can get to take the action we care about? And I’ll say, this number, when you’re just starting out, is usually the squishiest one.

If you haven’t run many tests before, you may not yet have a solid sense of how effective your different ideas will be. But you’ll get better at that with time. As you run more tests, you start learning what tends to work.Then your final step is the action conversion rate. Out of the people who take this initial action, how many go on to fully convert? That could mean buying, donating, subscribing, whatever your goal is.

Your North Star metric is the big outcome we’re all excited about. For example, maybe you work at a nonprofit, and you know something like, “Okay, around 30% of people who click on an email usually go on to donate.” So again, we’re just going to multiply all these numbers together.

That will give you your number of incremental conversions from the test. It tells you, “Hey, if this works as well as I think it will, I’m going to get 50 additional subscriptions, donations, or purchases.” Let’s try this with an email example.

Let’s say you’re an email marketer at a nonprofit. You’ve been using the same email template to send out donation appeals. And that template hasn’t been updated since, let’s say, the year 2010. It looks like someone copied a Word document and pasted it straight into your ESP. Now you’re thinking, “I’ve got a great opportunity here to improve the design of this email and increase our click-through rate.” You decide, “I’m going to move the call to action above the fold, and I’ll make the email mobile responsive.” You’re really excited. So now, let’s think: because of this new and improved design, how many more donations do we think we’ll get? Let’s start with the size of your audience. Say you’ve got 50,000 people you want to reach—50,000 people you’d like to donate to your cause. And based on past sends, let’s say you usually get about a 25% open rate. Great. That means around 12,000 people will open the email and see this new design—the RB variant.

Let’s say you have about a 2% baseline click-through rate on this email. Even though it hasn’t been updated in a long time, your supporters are passionate and really care about your cause. That means 98% of people are not clicking. Perfect. That gives us just over 12,000 people whose behavior we could potentially change with this fantastic new email design.

Now let’s ask, how many of those people do we think will click, now that the design is improved and it's easier to know where to go? You might say, I think I can get 1% of these people to click through. We'll go from 2% to maybe two and a half percent. Is that the math on that? That feels like a meaningful increase. Great. So that gives us 123 incremental clicks, 123 more people doing what we want them to do.

Now let’s say your click-to-donate rate is around 30%. That would mean 37 incremental donations. At this point, you can look at your average donation value and start calculating the ROI of redesigning this email. You can ask, does this make financial sense? Am I getting more out of doing this than I would if I didn’t make the change? That insight is useful even when you are looking at just one individual test. But it becomes even more helpful when you are comparing multiple tests and trying to decide, what should I do next? And for that, I’ll use an example from my time at Happier, the meditation app.

At Happier, we allowed a lot of people to sign up for a free trial of our meditation app on the web. But it was an app. You couldn’t plan meditations, learn anything, or really do much unless you downloaded and engaged with the app. We had a welcome email with a decent open rate, and it usually engaged about 40% of the people who were interested. One of the questions we asked ourselves was, what are the key actions people take that lead them to get value from the app and eventually subscribe? We knew they needed to open the app and play a meditation. There were a few other important actions too, but I won’t go into all of them now, since I’m focusing on these two.

We had a couple of hypotheses. One was, let’s make it as easy as possible for people. Let’s just focus on getting them to open the app. So we changed the first email. We made the app store download button big and clear. We made it as simple as possible for them to take that first step. But we had another idea too. What if we created an email that reminded people why they wanted to meditate in the first place? What if we helped them feel like we could teach them how to do this tricky thing? So we came up with a different version. We created a personalized meditation suggestion for them and focused on getting them to play that specific meditation.

We think this approach might help people engage more, remember the value we’re offering, and understand how we’re trying to help them. Ideally, that would lead them to subscribe later on.

So, let’s go ahead and start working through the ICE-T Framework for these two test ideas, and we’ll compare them. Let’s say we get 10,000 new people signing up for a free trial on the web every week. And let’s say, in general, about 50% of those people will go on to open the app. That means we have 5,000 people who are not opening the app—so they represent our opportunity for behavior change. We generally see about 40% of folks opening this email. That gives us an addressable audience of 2,000 people who haven’t done what we want but are paying attention. Perfect. Now let’s say we can get 10% of those people to open the app. Opening an app isn’t the hardest thing in the world, so that feels realistic. That gives us 200 people opening the app who otherwise wouldn’t have. Of those, about 15% typically go on to subscribe. Great. That gives us 30 incremental subscriptions. So if this test performs the way we expect, we can anticipate 30 additional subscriptions per week.

Now let’s compare our second test. We still have 10,000 people signing up per week. But fewer people actually go on to practice meditation. As a baseline, only 25% of people open the app, so we’re looking at a larger group of people who aren’t doing what we believe will help them the most. We still have the same 40% open rate, which gives us an addressable audience of 3,000 people. Meditating is more complicated than just opening the app. It’s not impossible, but taking that step is a bit harder. So I’m being conservative here and reducing our expected action rate a bit.

I think this one's going to be a little bit tougher, but that means we have 150 people who played a meditation who wouldn’t have done so otherwise. Now let’s say that if someone plays a meditation, they are more likely to subscribe. Around 60 percent of people who play a meditation go on to subscribe. That means we get 90 incremental subscriptions. So that’s three times as many incremental subscriptions from our meditation test idea compared to the app open test. So we’re done, right? We got it. We’re three times better. We’re feeling great. Just kidding. There is more to this than just impact sizing.

Impact sizing is important, but it is only one part of the ICE-T Framework. We already talked about impact, how much the test might move our North Star metric. Now let’s talk about confidence. How likely do we think this test is to succeed? There are many ways to build or lose confidence in a test idea. One of the things I often think about is, have we run tests like this before? Have we seen this metric move in response to past experiments? Has anyone else in the company tested something similar, maybe in the product or in another channel, and seen results? Do we have case studies or examples from companies like ours where this approach worked well?

I talk to users, and they often tell me, “This part of what you're doing,” or, “This part of your product was confusing.” When I hear that, I think, if I explain it better, that might give me more confidence that they’ll take the actions I care about. Once we have confidence, the next factor is ease. How much work is this test going to take? Changing a subject line is very different from redesigning an entire email template. That’s just the reality. So when you're thinking about the order of your tests and what to try, it helps to keep ease top of mind.

Then we get to time to learn. How long will we need to run this test before we can see a result? We all know, whether we like it or not, that interest compounds. Something similar happens here. If you get a win early, you benefit from that win for a longer period of time. You’ll have more subscriptions as a baseline for longer. That means your impact compounds. But if your test takes a long time to show results, it can be hard to prioritize, since you can only run so many tests at once.

Here’s how I tend to think about confidence. High confidence means I’ve done this before. It doesn’t guarantee the test will work, but I’ve seen a good result in this area. Medium confidence means other people have done it before. Maybe someone else in the company, or at a similar company, and it worked for them. Low confidence means I haven’t done it before, and I haven’t seen anyone else do it either. But maybe the idea is bold or exciting enough that we’re still interested in trying it.

Now let’s talk about ease again. I try to keep this simple. How much time will this take in actual work hours? You might have your own cutoffs, but for me, high ease means it takes a small amount of time. Low ease means it takes a lot of time. It might even mean blood, sweat, and tears. 

And finally, you need to think about your time to learn. There are two parts to that. First, how many users do you need in the test to determine whether it worked? Second, how long will it take to get that many users to the right stage in the lifecycle so you can actually run the test?

I don’t have enough time to go deep into a concept I’m going to reference here, but it’s included in the slides, and many of you probably already know it. It’s the idea of minimum detectable effect.

This is about answering a statistical question. How many people do we need in a test to detect an effect of 5 percent, 10 percent, 15 percent, or 20 percent? Let’s say you’re trying to improve a smaller metric, like increasing a click rate from 1 percent to 1.5 percent. That kind of test usually takes longer. You’ll probably need more people in that test than you would for a larger shift, like going from 10 percent to 15 percent. Even though both are a 50 percent relative improvement, the smaller the baseline metric, the more time and users it usually takes to detect a meaningful difference. That’s why this concept comes up when you’re calculating your minimum detectable effect. It can help you decide which metrics are worth focusing on. As I mentioned, there will be links later. Also, when it comes to calculating minimum detectable effect, there are plenty of great tools online. You don’t need to do this manually. I use a calculator at abtestguide.com. Now let’s talk about time to learn for the two test ideas I shared earlier from Happier, the meditation app.

First, the app open test. According to abtestguide.com, I need 3,000 users for this test. From my impact sizing earlier, I know about 2,000 people per week enter the addressable audience. That means it will take about a week and a half. Let’s round that up to two weeks.

Now let’s look at the played meditation test. Because the baseline action rate is lower, I’ll need more people in this test to measure whether I’ve had the impact I want. Let’s say abtestguide.com tells me I need 12,000 users. 

More people enter the addressable audience per week for this test compared to the app open test, but even so, it will take about four weeks to get enough users and results. So now we have a full ICE-T comparison between the two tests. We already know that the app open test, if it performs as expected, will lead to 30 incremental subscriptions per week.

I would rate my confidence in this test as high. I’ve done this before. I ran an SMS test to encourage more people to open the app, and it worked. So I feel confident this is something we can move through lifecycle marketing. In terms of ease, I’d say it’s relatively simple. I’m just updating the welcome email and focusing it on app opens. That won’t take much time to implement, and we’ll know in about two weeks if it worked.

Now let’s look at the played meditation test. It gives us more incremental subscriptions. Three times as many, in fact. In terms of confidence, I haven’t tested this in lifecycle before, but my product team has done a lot of tests aimed at increasing the number of people who play a meditation. So I have some confidence that this could work. But the ease is low. I want to create a really cool, personalized message with imagery. I might need to set up a catalog to reference the right content for each individual. So I know this will take more time to build, and the time to learn will also be longer.

Now, thinking back to what I mentioned earlier about compounding results, what should I do here? Even though the meditation email has a much bigger impact? I might decide to move quickly on the app open test. It takes less than five hours to set up. I can turn it on and let it run for two weeks. During those two weeks, I can spend time building the played meditation email, which is my bigger bet. Then I can launch that second test when it’s ready. Of course, it depends on your goals and how quickly you need to achieve them. But this gives you a clear way to make the decision yourself and also to explain that decision to anyone else on your team.

So what has this framework done for me? In the first six months or so at GlossGenius, we used this framework to choose a bunch of tests. Some of them failed. But some of them worked, which was great. The ones that worked helped us achieve an 8 percent increase in our overall subscription rate from lifecycle marketing alone. And I would love to tell this room that 8 percent is not the biggest number in the world. I know some people see higher. But if you’ve done lifecycle work, you know that moving your overall subscription rate by 8 percent is a big win. This framework also helped me make the case for prioritization across the company.

Before we had any wins, I needed help from my engineering team and some support from product to make these tests happen. Because I was able to say things like, “I have this test idea that could get us 100 incremental subscriptions per month,” it was easier for those teams to understand. That is the same kind of language the product team uses to prioritize their own tests. So it made it easier for them to see, “Here’s why this matters. Here’s why we should support what this new person is trying to do.”

I’m going to get a little philosophical as we wrap up. It’s a tense time in the knowledge economy for a lot of reasons. There are macroeconomic shifts, questions about the future, and uncertainty around how AI will change our workflows. With everything going on, I want to take a moment to talk about why the work we do matters. We don’t build products. We’re not PMs, and we’re not engineers. We don’t directly acquire customers either. Our colleagues in paid advertising, sales, and other go-to-market roles are the ones out there finding people who might want to use our product. In the nonprofit world, we don’t directly deliver services either.

You could say we’re the back of the house. But in reality, we’re hired to be force multipliers for acquisition, retention, and upsell. Across the entire funnel, we can improve the margins once a company has decided it has something people want. We are the ones who go out and tell the story in a way that pulls people in. That’s when lifecycle marketers tend to get hired. I think my friend Jacob put it best in a post he shared, and it aligns with what I often tell founders. People ask, "When’s the right time to hire for lifecycle?"

I usually respond with a question: "Do you have enough revenue? Is your business far enough along that a five percent win would actually make a difference?" If the answer is yes, then I’d say yes—it’s probably time, maybe even a little late. These marginal wins can be incredibly meaningful. They can shift the trajectory of a business.

But here’s the tension. When margins tighten, companies cut back. So if you were hired because a five percent lift mattered, what happens when five percent no longer matters? I still care deeply about marginal gains, and I think they’re important. But I also believe we need to be more than just the people working at the edges. We have a real opportunity to show why lifecycle matters beyond just direct revenue impact. We are close to the customer. If someone in the company has an idea, we’re the ones who can write the email or SMS, build a few experiments, and send them out. No engineering needed.

That means we can help de-risk ideas. We can test if people want a feature, a message, or a product before investing in building it. We can validate value without paying for ads or launching a campaign. That’s incredibly powerful. If we take on that role—if we lead experimentation in our organizations—we become an essential part of the growth team. We’re not just smoothing the process. We’re right at the front.

Take the example I shared earlier about the eight percent lift we saw from lifecycle efforts. When we ran that experiment, the insight that led to it was pretty simple. We realized it was actually kind of hard for people to pay for their subscription to our app. There was a lot of friction just putting down a credit card. We noticed that. We tested it within lifecycle and saw strong results. Then I took those results to my product team. I said, "Look, if we got an eight percent lift just from lifecycle tests, imagine what we could do if you went in, cleaned up the flows, and rebuilt the experience. I bet we could get an even bigger win."

We had an 8 percent win, which turned into a 20 percent win. That’s why I believe so strongly in our ability to be at the forefront of experimentation. We need to understand what matters and use that language when we do the kind of impact sizing I’ve been talking about. That is how growth teams work. It’s how product teams and finance teams work. And because that’s how companies operate, that’s why you’re here and what you’ve been hired to do.

Before I wrap up, I just want to say how excited I am to be here. One of my favorite quotes that I’ve heard recently is, "All models are wrong. Some are useful." The models I showed you today are definitely not perfect. But I hope they’re useful. If you have ideas for how to make these models less wrong, or if you have your own ways of thinking about this work that are even more useful, I would love to hear about them.

Please come find me and chat. Thank you so much.