Countering WordPress XML-RPC Attacks with fail2ban

In my last post I began inquiring into the WordPress XML-RPC attacks I’ve been sustaining here on the site. Since then I’ve been further studying the threat and experimenting with responses, and I have now developed working countermeasures and cast them into live operation. These countermeasures involve forwarding telemetry out of WordPress for pickup by the fail2ban facility, allowing for the detection and banning of attackers trying to exploit xmlrpc.php. Where other recommendations call for disabling affected methods or the whole XML-RPC subsystem, my more refined techniques control attacks while maintaining the full service set in operation for valid procedure calls.

My original assumption of the relationship between the excess XML-RPC request traffic I was seeing and password dictionary attacks turned out to be only partially correct. A component of the traffic is, in fact, dictionary attacking authenticating methods (like wp.getUsersBlogs), and these intruders can indeed be trapped via the wp_login_failed hook. But the majority of the traffic consists of a nonauthenticating type of request: forged pingback requests (calling method pingback.ping). In this attack, a command server tries to leverage my WordPress installation and thousands of others to mount distributed denial of service attacks against victim sites, as has been widely described. Meeting this attack was trickier as the WordPress core implementation goes out of its way to bury the relevant events.

The basic premise of signaling fail2ban from WordPress is not my original idea. It comes from the WP fail2ban plugin, which I’d initially set out to use verbatim along with the wordpress.conf filter that ships with it. The plugin uses PHP’s built-in functions openlog() and syslog() to communicate with the Unix system logger. It worked straight out of the box in my Ubuntu 14.04LTS test environment, but it gave us some trouble on CentOS production. After fighting with it for a while, I ultimately borrowed the lines of code I needed – with all due gratitude to the author of the plugin – and rolled my own adaptation. The principle stays the same, I just stripped it down to the parts I needed and then added some tricks of my own. I’m placing the code in my theme’s functions.php for now. I might package it up as a proper plugin later.

Anatomy of the XML-RPC password dictionary attack

In a dictionary attack, intruders attempt to gain unauthorized access by automatically throwing thousands of commonly used login credentials at an authenticating service, in hopes that any will work. Login names are often easy to guess (e.g. “admin”) or can be trivially enumerated, and unfortunately, people love to use extremely weak, well known passwords, so the approach has a high probability of success. This form of attack is older than the internet and is ongoing constantly against any and all exposed entry points of all varieties.

WordPress attackers would ordinarily direct requests against the wp-login.php page, and they do try to tirelessly, only to be rejected HTTP 403 as I have closed that exposure via an .htaccess IP address whitelist. (To be sure, the vast majority of self-hosted WordPress sites do not even take this step.) That brings them around to xmlrpc.php, WordPress’ XML-RPC endpoint, which offers around 60 different authenticating methods that they may choose from. Attacks are known to exploit wp.getUsersBlogs, I believe due to its compactness compared to many of the other calls, but perhaps for no other reason than it occurs topmost in /wp-includes/class-wp-xmlrpc-server.php.

An attack request can be simulated with the following short script:

#!/usr/bin/python

import xmlrpclib
proxy = xmlrpclib.ServerProxy("http://www.yoursite.com/xmlrpc.php")
print proxy.wp.getUsersBlogs("baduser","badpass")

For valid credentials this will return a response containing some XML-RPC structured data, and that’s what the attackers are looking for. For bad credentials the returned response will contain an XML-RPC Fault 403 object. In both cases the web server’s access_log will record the transaction as having been completed HTTP 200 OK.

This brings me to the first security deficiency I wish to bemoan about the XML-RPC implementation. Error events always get buried in the response content body, with the HTTP status reporting 200 no matter what actually happened. If the HTTP response status was set to a meaningful error code, anything but 200, we might be talking about triggering countermeasures off access_log solely and not messing around with the application layer. It turns out that it’s implemented this way to comply with the XML-RPC protocol (rather than to frustrate security engineers) and is idiosyncratic to XML-RPC, not WordPress. I bemoan it either way.

Trapping the password dictionary attack

We can trap these (actually all) login failures relatively painlessly via WordPress’ wp_login_failed hook. In my research I noted some uncertainty as to whether the hook worked properly from the XML-RPC context. I have tested it conclusively and can confirm that it works just fine. Here is a snippet of the trapping logic:

add_action( 'wp_login_failed', 'fail2ban_login_failed_hook' );
function fail2ban_login_failed_hook($username) {
   openlog('wordpress('.$_SERVER['HTTP_HOST'].')', LOG_NDELAY|LOG_PID, LOG_AUTHPRIV);
   syslog(LOG_NOTICE,"Authentication failure for $username from ".$_SERVER['REMOTE_ADDR']);
}

As advertised this is boiled down straight from the WP fail2ban plugin and functions identically. On a wp_login_failed authentication failure event, we’re queuing a message through the system logger facility with NOTICE priority to LOG_AUTHPRIV for subsequent pickup by fail2ban. The message includes the IP address of the offender for fail2ban’s use. It will be recorded on any authentication failures including those from wp-login.php, from xmlrpc.php, and from anywhere else that calls the core wp_authenticate function now or in the future.

LOG_AUTHPRIV corresponds to a log file in /var/log/ contingent on your distribution’s implementation of syslog.conf. On my Ubuntu test server it means /var/log/auth.log. On CentOS production it means /var/log/secure. You’ll need to figure out which one, not least of which so you’ll know where to tell fail2ban to look later.

Anatomy of the XML-RPC forged pingback attack

The other type of XML-RPC attack traffic I’m seeing involves third party pingbacks, using method pingback.ping to flood victims. Thankfully, I’m just the third party, not the victim.

In this attack, valid looking XML-RPC pingback requests are forged to appear to hail from a chosen victim and posted to thousands of WordPress sites. These thousands of “helper” sites unknowingly participate in the attack by dutifully GETting the target page on the victim’s site for the purposes of verifying the supposed backlink, just as they’ve been programmed to do. The result is a distributed denial of service attack on the victim, without amplification, but with the attacker having achieved indirection to disguise his identity (though not very well) and complicate countering.

An attack request can be simulated with the following short script:

#!/usr/bin/python

import xmlrpclib
proxy = xmlrpclib.ServerProxy("http://www.yoursite.com/xmlrpc.php")
print proxy.pingback.ping("http://www.victim.com/some/valid/resource/","http://www.yoursite.com/some/pingbackable/resource/")

The pingback logic in WordPress won’t perform the GET request to victim.com for just a base domain URL, it bails for that case, so the attack must use a path to some actual page underneath to be effective, and let’s assume the larger the better. It also must purport an actual pingback-able page on the helper site as this gets verified before initiating the request.

Now I know you’re thinking, so what, the attacker can trick a server into sending one measly HTTP GET request, big deal. Well, now imagine this attacker has a botnet of initiating command servers and a prebuilt, preverified library of 162,000 helper sites at their disposal, and can run traffic through a good fraction of those round robin, say, every three minutes, against a single victim site. For the victim it would be very difficult to keep their network from folding, no less their site up and running. Notice also that the attacker never has to compromise the huge collection of helper sites. All they’re doing is accessing legitimately purposed, default enabled, weakly protected open RPC services.

As an unwitting third party helper node in this scheme, all you see in access_log is a series of requests to xmlrpc.php at regular intervals returning, again, status 200. But under the hood, a lot of hidden activity takes place. Module /wp-includes/class-wp-xmlrpc-server.php has all this logic and it’s surprisingly ugly for core. WordPress validates the source and target URLs, tries to map the target to a pingback-able post on its own site, checks for duplication, makes the GET request for the source page, and if it can get it, picks it apart looking for the title, the claimed backlink and a surrounding excerpt. When any of this goes horribly wrong, the logic will bail and throw an XML-RPC Fault response. I’ll come back to this point in a moment.

Trapping the forged pingback attack

The WP fail2ban plugin‘s solution for trapping pingback attacks taps into WordPress’ xmlrpc_call hook, which fires with a parameter of pingback.ping on entry of the process just described. Trapping logic functionally identical to the plugin goes like:

add_action( 'xmlrpc_call', 'fail2ban_pingback_hook' );
function fail2ban_pingback_hook($call) {
   if ('pingback.ping' == $call) {
      openlog('wordpress('.$_SERVER['HTTP_HOST'].')', LOG_NDELAY|LOG_PID, LOG_AUTHPRIV);
      syslog(LOG_NOTICE,"Pingback requested from ".$_SERVER['REMOTE_ADDR']);
   }
}

This writes a message through the system logger to LOG_AUTHPRIV with the IP address of every pingback requester for subsequent pickup by fail2ban, in similar fashion to what we rigged against the wp_login_failed hook for authentication failures above.

This works OK of course, but by triggering on every xmlrpc_call(pingback.ping) it makes no distinction between valid and rogue pingback requests. In effect the responsibility for distinguishing between harmless and malicious pingback traffic is deferred downstream to fail2ban’s ban engine, with very high potential for false positives.

The realization came to me that all rogue pingback requests must error out during the validation process. This must be true, if not for any of the multitude of possible other reasons, due to the fact that the victim page never actually contains a valid backlink. The whole thing is forged, remember. Studying the code, function pingback_ping in /wp-includes/class-wp-xmlrpc-server.php throws a whole series of different error codes and messages, including yes, error 17, “The source URL does not contain a link to the target URL, and so cannot be used as a source.” And, heaven bless you core WordPress developers, they’re all filterable via hook xmlrpc_pingback_error on their way to becoming XML-RPC Fault object responses.

Setting out to trap xmlrpc_pingback_error, I discovered the second security deficiency I wish to touch upon. The default xmlrpc_pingback_error filter (defined in /wp-includes/comment.php) inexplicably strips the error details in almost all cases, except for code 48, corresponding to the duplication case. I had to pull my filter forward in priority to get ahead of the default stripping. You’ll virtually only ever get a status 200 and an XML-RPC Fault object body with code 0 and an empty error message out of WordPress, no matter what actually happened while processing a pingback request. This annoyed me for a minute during my testing. I imagine there’s a perfectly good rationale but I was surprised to see core go so out of its way to withhold these error codes.

Here is the final trapping logic I’m rolling with for pingbacks:

add_filter( 'xmlrpc_pingback_error', 'fail2ban_pingback_error_hook', 1 );
function fail2ban_pingback_error_hook($ixr_error) {
   if ( $ixr_error->code === 48 ) return $ixr_error; // don't punish duplication
   openlog('wordpress('.$_SERVER['HTTP_HOST'].')', LOG_NDELAY|LOG_PID, LOG_AUTHPRIV);
   syslog(LOG_NOTICE,"Pingback error ".$ixr_error->code." generated from ".$_SERVER['REMOTE_ADDR']);
   return $ixr_error;
}

As distinguished from the original plugin’s trap, this more selective trap will only fire when something not right about a pingback request causes regular processing to halt and exit. It will never fire during processing of a legitimate pingback that ends successfully, nor on an accidentally submitted duplicate of an already registered one. This is a material distinction, because it’s going to let fail2ban decide “has this IP address caused an excessive frequency of pingback errors” as opposed to the thoroughly ambiguous and false positive prone “has this IP address posted excessively frequent pingbacks of any kind”.

My enhancements also include capturing the error code to the log facility for administrator reference. The 1 in add_filter() pulls forward the priority to evade stripping of the fault details for this purpose.

Dropping the banhammer

With the above traps implemented on your WordPress site, and either through sustaining real world attack traffic or by using the sample scripts provided to simulate attack requests, you should see event notices making their way into LOG_AUTHPRIV. Now it’s time to get fail2ban to pick up these messages so the banning can begin.

Obviously you’ll need to install fail2ban to your environment if you haven’t already. I won’t cover the installation and basic usage here as it’s documented elsewhere. The distributions I work with have made packages readily available for as long as I’ve been working with it. Out of the box it will generally have an sshd jail enabled by default along with numerous other modules present but disabled.

Fail2ban is an extremely versatile agent. Almost too versatile. It works by monitoring system and application logs for events using fully configurable regular expression based filters, and sensing the IP addresses of offenders in these log lines. When an IP address has transgressed too many times in a given time span, it gets thrown in “jail” for a period – all configurable values – the primary consequence of which (though additional actions are also possible) is to establish a DROP rule via the Linux kernel’s rules-based firewall interface, iptables. Further network traffic from the jailed IP address will then simply be dropped for the remainder of the period, before it can even touch the system, cutting off whatever nonsense they were up to “with extreme prejudice,” as the saying goes.

To get this working we’re going to need to install a custom filter and define a custom jail, using the following process.

Because I disrespected the log message format of the WP fail2ban plugin‘s implementation when introducing some of my enhancements, unfortunately the wordpress.conf fail2ban filter that ships with it can no longer serve without modification. I have adapted it to accommodate my new pingback error line. Here is the adapted wordpress.conf modified to receive messages from my two traps. I reiterate my thanks to the author of the original plugin whose work I’ve modified.

# Fail2Ban filter for wordpress
#

[INCLUDES]

before = common.conf

[Definition]

_daemon = wordpress

failregex = ^%(__prefix_line)sAuthentication failure for .* from <HOST>$
            ^%(__prefix_line)sPingback error .* generated from <HOST>$

ignoreregex =

# Author: Scott Brown

Place this at /etc/fail2ban/filter.d/wordpress.conf alongside the other filters found there.

Now we need to set up the jail by placing some directives in /etc/fail2ban/jail.local. Use your discretion on the values of maxretry, findtime, and bantime, bearing in mind that you’re either inheriting or overriding defaults that may or may not be set in the primary jail.conf in the same path. Distributions seemed to differ in their packaging, so plan to verify and test your settings rigorously to make sure you get the behavior you expect.

As for me, I wanted fail2ban to integrate for an hour, and ban offenders for 24 hours. (This may likely get tweaked in the future depending on how it performs.) On my Ubuntu test environment, jail.local got:

[wordpress]
enabled = true
port = http,https
filter = wordpress
logpath = /var/log/auth.log
findtime = 3600
bantime = 86400

While in contrast, on CentOS production, jail.local got:

[wordpress]
enabled = true
filter = wordpress
logpath = /var/log/secure
findtime = 3600
bantime = 86400
action = iptables-multiport[name=wordpress,port="80,443"]

In both cases a default maxretry value of 3 came from upstream. Note the difference of logpath corresponding to each distribution’s variation of LOG_AUTHPRIV as discussed earlier. The other difference concerns the action. Ubuntu’s packaging of fail2ban’s primary jail.conf defines some macros for default actions so that they don’t have to be restated here. CentOS’ does not, hence we have to make action explicit.

Once you put these changes in place, either bounce the fail2ban service or issue sudo fail2ban-client reload to force it to reload its configurations.

Results

These countermeasures have been working basically as expected so far, in less than a week of operation. The pingback error trap has shown the greater effectiveness of the two types. Pingback attackers were hitting me at high frequency from just a handful of IP addresses, making themselves easy to catch. That activity now gets cut off at 3 attempts within minutes of activating the jail.

The authentication dictionary attack, however, appears to arrive from a myriad of IP addresses, a different host each attempt. This interprets to the attack traffic originating from a massive botnet or nets or just consisting of scans from random attackers on the internet at large. At any rate fail2ban isn’t made to counter such a distributed attack. It catches the failures but no individual IP address ever exceeds maxretry, over the integration period I’m using anyway. But this was the smaller fraction of xmlrpc.php attack traffic.

The test I’m interested in will be to watch what happens to this chart in the coming months:

I expect to see served xmlrpc.php traffic drop back down to the tiny fraction of page views at which it stood before these waves of attacks got underway. I’ll report back in a month or two.

Resources

SAOTN: Huge increase in WordPress xmlrpc.php POST requests

Sucuri: More Than 162,000 WordPress Sites Used for Distributed Denial of Service Attack

fail2ban Homepage

WP fail2ban plugin