Sometimes, Throwing Hardware At A Performance Problem Is A Bad Idea.
Multithreaded execution can squeeze every last drop of performance out of these resources.
It is a common problem in a growing business: IT infrastructure starts getting slow due to rapidly growing demand for the services it provides. More customers, more employees, more data records, and more network traffic will ultimately mean a bottleneck somewhere. The solution is most often framed as a choice between buying more hardware or paying developers to optimize your application to do its work more efficiently.
Sometimes, paying for more and/or better hardware is a no-brainer.
Like when your dedicated server is really old or when your server is low-cost starter model or otherwise has lackluster specs. If your server has only a single-core processor, a CPU speed of 2.0Ghz or less, or less than 1GB of RAM, you could probably get considerably more life out of your application by up-sizing the hardware that runs it.
There are other situations, though, where it’s not such a good idea.
If your server already has multiple cores, lots of RAM, and a high clock speed, then it probably won’t require too much growth in demand until your server has reached the limits of commonly available servers. If this is the case, you should probably start to think about ways to modify your application to run on a cluster of machines rather than a single machine. It’s usually pretty easy to separate database duties to a separate machine. If you find yourself facing this situation, Boomcycle is likely to be very helpful!
And then are some circumstances where more hardware is a really bad idea. We were contacted a few years back by a client that had a particular problem with an asset pipeline. His server would receive data feeds from various partners each night and, once these data feeds were imported to a database, the server would begin fetching and processing thousands of image assets to accompany the imported data.
These images, of all shapes, sizes, and types, had to be imported and processed to consistently conform to our client’s site design and performance standards. A script was created for downloading these images, rendering thumbnails of varying sizes, and delivering the re-sized images to a Content Delivery Network (CDN) for hosting. The script, although well-designed, faced a fundamental problem: request latency. Each time the script would request a remote image, it might have to wait a second or so for the remote server to actually cough up the image — while that one second elapsed, the script was idle as code execution blocked waiting for a response. There was also a request latency issue as the script waited for the CDN to accept the thumbnails generated by the script. This script, only one thread of execution, would run constantly for hours on the server yet the server was completely idle over 90% of the time while it waited on responses from remote machines.
Throwing more hardware at this problem would have been a huge waste of money.
Were we to allocate a giant, expensive beast of a server, this machine would still be at the mercy of request latency, and would probably offer no performance benefit whatsoever while doubling or trebling server cost.
Our solution to this problem was to alter the script responsible for processing this asset pipeline so that it would spawn off a separate process to handle each image. Multithreading — or more precisely multiprocessing in this case — is an ideal strategy when you have a lot of work that can be done in parallel but have to wait for I/O in order to complete any single work task. The idea is that you spawn off a swarm of processes to make many simultaneous requests for the remote images and then, as the requests trickle back from the many remote servers, you queue them up to be processed.
Because you make many simultaneous requests rather than one-by-one sequential requests, you end up with much greater throughput on the machine. The result is that the machine is constantly busy processing images. In fact, it’s very easy to reach the limitations of the machine by simply spawning off more simultaneous processes.
Although this multiprocessing approach resulted in near-perfect utilization of the machine dedicated to the asset pipeline — even when we scaled up to a more powerful server — it was still not enough to satisfy the growing needs of our client. Fortunately, the software design that we implemented to support multiprocessing also made it easy to add additional machines for processing the asset pipeline. In fact, adding these was so easy that we began to overwork our database server. A few tweaks helped to get the database server back into the sweet spot.
However there is a high cost to having multiple managed servers constantly running to handle a periodic need for enormous amounts of processing power. These import processes would run only twice a day and it was important to process all of the images in the pipeline *quickly* so that the data was made ready quickly. We needed as many as 10 servers to get this processing done in the necessary time frame but still had a situation wherein our servers were idle 90% of the day. Our client was still paying for a lot of computing resources that were simply not being utilized.
Boomcycle’s solution in this case was to develop logic to access the Rackspace Cloud API so that these servers could be allocated when needed and then shut down once there work is done. We also developed custom analytics so that our client can monitor the number of images to be processed, the number of servers currently online, etc. These analytic samples, taken at five minute intervals, showed us that on average, we had only 1.08 servers processing the asset pipeline at a given time. 1.08 servers processing is considerably less than the 8 that we formerly needed at peak hours when our autoscaler fired up the extra horsepower to get the job done in the desired time frame.
Given that each of these servers costs about $230 per month to run, you can see that we have effected considerable cost savings. Rather than running 8 servers constantly at a cost of $1,840 per month, we run on average 1.08 at a cost of $248.40 per month. At this point, the disparity amounts to over $19,000 annually. We expect that as their processing need grows that this disparity will become even greater, saving our client even more money.
Sometimes, writing better software really pays off.