Last quarter, newsletter giant The Hustle ran a test that earned 43,876 additional clicks on their emails.
Swimming 44,000 total. 44,000 more.
If you were like me, you would do terrible things for that kind of engagement.
So I knocked on the door of their editor-in-chief and demanded to know his secrets.
Fortunately, the process is simple and repeatable—no scary stuff required. So put down that crowbar, because I’m going to share a process you can start today.
How to do it
Ben Berkley, editor-in-chief of The Hustle magazine, is constantly smiling wryly. Which is fitting, because behind that smile is a deceptively simple method—with some twists to come.
Here’s how it works:
Each weekday morning, Ben prepares two separate headlines for each newsletter. At 5:30am ET, each item is sent to only 30% of The Hustle’s audience.
After an hour, the variant with the highest engagement is automatically sent to the remaining 40% of recipients.
“That’s it?” you are thinking. “It’s just a stinking A/B test?!”
Swim so fast. I promised you trades:
- Variations are sent to four predefined subscriber segments.
- The winner is based on click through rate (CTR) — not the open rate.
- Ben’s Secret Sauce: The WTF Factor.
Let’s dig in.
Turn 1: Segmenting readers
Hustle shares the original two test emails four segments.
To help me figure this out, Ben refers me to Kaylee Jenzen, the MarTech wizard at Hustle.
(“If the Nobel Foundation ever adds a newsletter award, I’m nominating Kaylee as soon as possible,” he says.)
“Hustle benefits HubSpot Workflows segment contacts with values ranging from 1 to 4,” explains Jenzen. “Email 1 targets segments 1 and 2, while Email 2 targets segments 3 and 4.”
In other words: one email, two variants, four sends. the reason?
“By distributing emails over time, The Hustle avoids triggering spam filters and ensures more consistent delivery rates,” says Kaylee.
takeaway: High sending volume is a signal that email service providers use to identify spammers. By reducing the recipient list and increasing the sending time, you make your email more secure in the eyes of the email server.
Twist 2: Target CTR, not open rate
If you’ve tried A/B testing your themes before, I bet you based your test on the open rate, right?
Almost any tutorial will tell you that, and it’s not a bad choice. But that’s not it only choice.
While the open rate measures the number of readers who access an email, it cannot separate out those readers who delete it immediately. Click-through rate, on the other hand, measures the number of readers who actually engage with the email.
“By emphasizing CTR, The Hustle aims to not only capture the reader’s attention, but also encourage them to take action and interact with the content,” says Jenzen. “This approach reflects a strategic effort to encourage meaningful engagement beyond the opening itself.”
You see, The Hustle’s open rate is already above the industry average, so tweaking the theme only results in about a 1% difference in that metric. (#HumbleBrag?) Instead, his team is making a strategic choice to focus that time and effort on cultivating even deeper engagement.
takeaway: Open rate power be the metric you want to focus on. But think about your ultimate goal. Optimize to achieve thatnot what the YouTube tutorial told you to do.
Twist 3: The WTF factor
“When people go through their inboxes, they’re often just looking to delete all the new stuff,” says Ben. “And they get into a rhythm.”
That beat is the sound of your email being deleted.
“So the goal is to gently throw someone out of their rhythm,” he says.
That’s where Ben’s secret sauce comes in.
“To do that, you need a little ‘WTF?’ oral factor.”
As the former executive editor of The Onion, Ben is no stranger to the WTF factor. But that doesn’t mean he’s trying to recreate the Onion titles for The Hustle.
“I’m never looking for shock value or trying too hard to throw people off,” he cautions. “But eventually you want them to see something unexpected and give them a quick break. You just want them to break for a split second.”
Some of my favorite examples:
- 🎨 Believe in yourself, but don’t like this
- 🕹 Buried under cement at the landfill
- 🔓 Toothpaste Jailbreak
takeaway: Your subject line should be surprising enough to get them off autopilot and take a few minutes for your email.
But what does that actually mean?
How to write cover lines like in a crowd
Ben shares with me what he calls his “Subject Line Manifesto.” These are the actual guiding principles for The Hustle’s editorial voice.
1. Be unexpected.
You’re unlikely to throw someone out of their groove with a topic that confirms something they already know.
“Having worked in comedy, I know the power of subverting expectations,” says Ben. “It’s basically about having a laugh and creating something memorable.”
“Excluding all other actual unforgivable crimes, there is only one unforgivable crime in The Hustle,” he adds. “Being boring.”
That doesn’t mean you have to try to be funny. In fact, it may be inappropriate for your brand. You can subvert expectations with new data, surprising ideas, or a thoughtful question.
What part of your email adds something new to the conversation? That’s your topic.
Speaking of which…
2. Strong themes come from strong stories.
“The subject matter must match the overall atmosphere of the newsletter, as well as the specific content of each day,” explains Ben. “Indeed, the work of writing strong subject matter begins with the work of writing strong stories.”
So if your email no add anything new to the conversation, maybe pause subject lines for now and revisit your content strategy.
3. Don’t go for the cheap laughs.
If you decide to use humor, don’t use the same pun that 10 of your competitors used. Chances are the example just popped into your head. Skip it.
“The low-hanging fruit in comedy is low-hanging fruit for a reason; it’s there because people expect it,” adds Ben. “If you give it to them, you might get a chuckle, or a pity party or two, but you probably won’t get a belly laugh because they weren’t surprised. Valuable content comes from recoil.”
And while we’re on the subject…
4. Never fall for clickbait.
“It’s so easy for a topic to feel like clickbait, and that’s the last feeling we want our audience to have,” warns Ben. “In this hyper-competitive attention economy, if you break that connection, you’ll never get it back.”
5. Keep it short.
Hustle sees a noticeable increase in performance from titles that are only 11-15 characters long.
Their shortest topics have almost the same engagement rate 75% more but the emails with the longest.
“As I approach the subjects, I always remind myself: those words are there to encourage the palate, not overwhelm it,” says Ben. “You have to give them enough context to keep them interested, but not so much that they feel like they already have the whole story.”
Now that you’re a subject matter expert, let’s talk about the process.
How to take the test at home
Sending multiple emails every day sounds like a lot of work, and it is would be if you did it manually. Fortunately, most email marketing tools have some sort of automated A/B testing.
He happens to be using Hustle Marketing Hubso we’ll show you how to do it within HubSpot, but the process should be pretty much the same no matter which tool you use.
1. Create your email as usual.
In HubSpot, as with most tools, this will automatically be the AA/B version of the test. (So name it accordingly and give it your first variation on the theme.)
2. Click “Create A/B Test” in the upper left corner.
3. Enter a name for version B, then click “OK”.
Don’t think about this. Naming conventions are one place where it’s safe to let your boring side show.
Even Ben forgives it in this picture below.
4. Edit the theme for version B.
Remember that the best practice for A/B testing is to change only one element per test. Otherwise, you risk distorting the results.
5. In the version drop-down menu, click “Test Management”.
6. Set the A/B distribution to the desired mix.
Hustle uses 30% of its recipients for each variation, with 40% reserved for the winner. But with over two million subscribers (#HumbleBrag verified), it can spare a few readers to get more accurate results.
If you have a lower sending volume, you may want to stick to 20-25% for each trial. While your test will be based on a smaller sample, a proven winner will be seen by more recipients.
7. Set the winning metric to “Click Travel rate.”
To replicate The Hustle method, you’ll want to choose CTR. Additionally, think about the goals of your campaign.
The tool will also allow you to choose an “open rate” or “click rate”.
8. Set “Test duration” to 1 hour.
You will see a warning to set a duration of “at least 4 hours” to achieve final results.
That’s good advice for most marketing emails, but it can be a problem for newsletters. HubSpot research shows that the best thing ever are between 9:00 AM and 12:00 PM ET.
You can see why a four-hour test might make it complicated. But what you’ll lose in statistical significance, you’ll make up for in eyeballs.
9. Click “Save Changes.”
10. Review your email settings and recipients as usual, then send.
Get involved
To reassure our legal department and prevent you from falling for it mine door: I can’t promise you 43,876 extra clicks.
But if you follow a data-driven approach, you’re almost guaranteed to see improvement. Couple that with Ben’s tips and you’ll be throwing your readers out of their groove on a regular basis.
At the very least, you’ll never be bored.
https://blog.hubspot.com/marketing/how-the-hustle-got-43876-more-clicks