Making Concurrent cURL Requests Using PHP’s curl_multi* Functions

The cURL library proves a valuable resource for developers needing to make use of common URL-based protocols (e.g., HTTP, FTP, etc.) for exchanging data. PHP provides a set of curl* wrapper functions in an extension that nicely integrates cURL’s functionality.

When you have to make multiple requests in a script, it’s often more efficient to utilize the curl_multi* functions (e.g., curl_multi_init), which make it possible to process requests concurrently. For example, if you have to make 2 web requests in a script and each one requires 2 seconds to complete, making 2 separate curl requests, one right after the other, requires 4 seconds. However, if you make use of the curl_multi* functions, the requests will be made concurrently (i.e., we no longer have to wait for one request to finish to start the next one), and requires only 2 seconds (the actual execution time depends on if the scripts are truly running in parallel or merely concurrently.)

Let’s take a look at a function that provides a simple interface to the concurrent capabilities of cURL and is extensible to most situations, as the curl_multi* functions can be cumbersome.

/**
* Simple wrapper function for concurrent request processing with PHP's cURL functions (i.e., using curl_multi* functions.)
*
* @param array $requests Array containing request url, post_data, and settings.
* @param array $opts Optional array containing general options for all requests.
* @return array Array containing keys from requests array and values of arrays each containing data (response, null if response empty or error), info (curl info, null if error), and error (error string if there was an error, otherwise null).
*/
function multi(array $requests, array $opts = [])
{
    // create array for curl handles
    $chs = [];
    // merge general curl options args with defaults
    $opts += [CURLOPT_CONNECTTIMEOUT => 3, CURLOPT_TIMEOUT => 3, CURLOPT_RETURNTRANSFER => 1];
    // create array for responses
    $responses = [];
    // init curl multi handle
    $mh = curl_multi_init();
    // create running flag
    $running = null;
    // cycle through requests and set up
    foreach ($requests as $key => $request) {
        // init individual curl handle
        $chs[$key] = curl_init();
        // set url
        curl_setopt($chs[$key], CURLOPT_URL, $request['url']);
        // check for post data and handle if present
        if ($request['post_data']) {
            curl_setopt($chs[$key], CURLOPT_POST, 1);
            curl_setopt($chs[$key], CURLOPT_POSTFIELDS, $request['post_array']);
        }
        // set opts 
        curl_setopt_array($chs[$key], (isset($request['opts']) ? $request['opts'] + $opts : $opts));
        curl_multi_add_handle($mh, $chs[$key]);
    }
    do {
        // execute curl requests
        curl_multi_exec($mh, $running);
        // block to avoid needless cycling until change in status
        curl_multi_select($mh);
    // check flag to see if we're done
    } while($running > 0);
    // cycle through requests
    foreach ($chs as $key => $ch) {
        // handle error
        if (curl_errno($ch)) {
            $responses[$key] = ['data' => null, 'info' => null, 'error' => curl_error($ch)];
        } else {
            // save successful response
            $responses[$key] = ['data' => curl_multi_getcontent($ch), 'info' => curl_getinfo($ch), 'error' => null];
        }
        // close individual handle
        curl_multi_remove_handle($mh, $ch);
    }
    // close multi handle
    curl_multi_close($mh);
    // return respones
    return $responses;
}

To use this function, you can call it like so:

$responses = multi([
    'google' => ['url' => 'http://google.com', 'opts' => [CURLOPT_TIMEOUT => 2]],
    'msu' => ['url'=> 'http://msu.edu']
]);

And, then you can cycle through the responses:

foreach ($responses as $response) {
    if ($response['error']) {
        // handle error
        continue;
    }
    // check for empty response
    if ($response['data'] === null) {
        // examine $response['info']
        continue;
    }
    // handle data
    $data = $response['data'];
    // do something extraordinary
}

While the above function is helpful for a few requests, if you need to make a large number of requests (perhaps more than 5), then instead you should have a look at the rolling curl library, which makes better use of resources.

And your significant other said you couldn’t multitask ūüôā

Fall-Through Functions

When embracing functional programming principles in languages that aren’t designed specifically for functional programming, dealing with side effects requires great care and discipline. For those who can’t remember what side effects are, side effects are attempts to modify the state of the world (at least in terms of the scope of your program’s environment) through a means other than the return value of a function (e.g., performing a SQL insert, printing text to the standard output device, sending an email, etc.) Hopefully you noticed the word “attempts.” The problem with trying to directly modify the state of the world is that you don’t know what state the world is in: sometimes we’re caught by surprise.

I try to compartmentalize side effects in functions that lack branching constructs (e.g., if/then, switch, etc.) I refer to this type of function as a¬†fall-through function¬†because the function proceeds line-by-line until it reaches the last line and returns the status or result.¬†Here’s a simple example of a fall-through function in PHP that sends an email:

<?php

$mail = function($to, $subject, $message){
    // handle longer lines
    $message = wordwrap($message, 70, "\r\n");
    // send message, returning status
    return mail($to, $subject, $message);
};

?>

Fall-through functions provide clean separation of the logic we want to test from the world-dependent states that are unpredictable (i.e., code containing side effects), and as we know, clear boundaries are a good thing.

My Dream of Mom

A Long Year

Mom and dad and Matthew and me in front of Grandpa and Grandma's home.

I normally reserve this blog for technology-related posts, but I felt it was fitting to preserve these particular personal memories.

My mom passed away last October. In January my maternal grandmother passed away. And, since January, my father has been living at our house on hospice for end-stage prostate cancer. Long year.

My father’s presence has been a blessing in many ways. My family lived the farthest from my parents’ house, and phone calls a few times a week only went so far to bridge the gap. I’d often longed for the opportunity for my parents to be more integrated into my children’s lives, and having my father here to interact with, enjoy, and love my daughters is something for which I’ll always be thankful. Sarah (my little 2 year old) has gotten the biggest kick out of walking to grandpa’s room and showing him the activity that currently holds her attention (broadcasting her voice at progressively higher volumes if necessary to gain his attention.)

That said, I must confess there are things that I haven’t considered blessings. I haven’t been a fan of catheter care, bathroom breaks, and a host of other things related personal grooming tasks that don’t involve me grooming myself (although I’ve gained great respect for the work of nurses and CNA’s in the process.) I haven’t enjoyed becoming “THE POLICEMAN” who tries to regulate the unfathomable combinations of food my father wanted to “enjoy” (he knew he’d pay for a hot dog dressed with onions and peanut butter, but his perseverance ¬†eventually caused me to cave.) And, most of all, I haven’t enjoyed seeing his capabilities and appearance diminish.

I know what you’re wondering: What’s all of this have to do with a dream about my mom? We’ll get there, just keep reading so I can provide a bit more backstory.

When Can We Leave?

There were things I didn’t enjoy that were especially associated with my mom. At church my mom was involved with many different tasks and responsibilities. She taught Sunday School, junior church, sang in the choir, hosted and scheduled coffee hours, directed Christmas pageants… well, I could go on, but you get the idea.

Now, her being involved in all of these tasks, that didn’t bother me a bit. What bothered me was having to wait for what seemed like an eternity to leave church every Sunday (and every other time we were there.) There was always something to work on, and this always meant that we (the “I” component here is what proved most painful) had to help out on many tasks if we wanted to attempt to get out of church before the service started for next week. These tasks often involved craft preparation of some sort, and let’s just say that to this day you’re not going to ever find my at Michaels. ¬†DIDN’T MY MOM REALIZE I WAS MISSING THE ONLY NBA BASKETBALL I COULD WATCH ON TV!!! Things were so dire in terms of our departure time from church, that I was given the responsibility of locking up the church, something successively passed down to the oldest Richardson not away at college (except for then-rebel Kathleen.)

The Dream

About three weeks ago, there was a day where I felt especially drained because of all of the tasks required to care for my father. I wasn’t proud of it, but I must confess I resented doing the work. I was short with my dad that evening, and I went to bed that night carrying a heart full of frustration. A couple nights later, I had a dream about mom, one I’ve not described to anyone until now.

I remember being in one of the classrooms of my old church. I remember the fluorescent lighting, the basement smell, the rough texture on the walls, and I even noticed the old carpet pattern. It was a Sunday, and my family was the only one left at church. Out of the corner of my eye, I could see someone turned away from me quietly standing at a table and working. It was mom! I felt a rush of excitement overcome me and I ran as fast as I could to her side. She turned to see me with her smile of contentment, and I truly can’t describe the feelings I had at this moment. It really felt like I was in my mom’s presence again, and I couldn’t wait to hear her first words to me. I, with the utmost of eagerness, exclaimed, “Mom, how can I help!” I truly longed to work beside her, whatever the task might be.

Before she could utter one word in response, I felt the dream start to unwind. I pleaded with God in my heart for the dream to continue. In that moment I can honestly say that it had seemed so real and I had been so hopeful for even the smallest interaction, that it felt like I was losing her all over again. I woke balling uncontrollably, and I’m crying now as I remember this emotively overwhelming memory.

God granted me a wonderful truth through this experience that has greatly impacted me these past few weeks:

There will come a time for each of us when once-detested tasks would be counted as treasures if we could spend one more moment with the one we love. Cherish every moment.

My moments with my wonderful father are fleeting fast.

Long Live the GOTO Statement

Introduction: Infamous GOTO

Sure, since Dijkstra’s letter outlining the harmful aspects of the goto statement, few have voiced even modest amounts of tolerance for the statement, let alone condoned it’s use. Even those who’ve described practical uses of the goto statement have questioned its existence in higher level languages (e.g., although¬†Donald Knuth noted some utility for goto, he also suggested that he would likely never use it in a language that had sufficiently capable iteration and event constructs.)

Today you can find a myriad of online resources that set the goto statement ablaze. The 5.3 release of PHP provided a unique look at the perception of the goto statement, as prior to that release, PHP lacked Continue reading Long Live the GOTO Statement