The (IT capacity) testing times we live in: Survey’s in – and my, how some of you have grown – The Register

Reg Reader Survey We decided to look at the science – or perhaps the art – of capacity management for our recent Reg Reader Survey.

We figured it was likely – particularly because of the radical changes caused in behaviour, by COVID-19 lockdowns, with the use of IT systems.

So we asked readers to tell us about how they’ve dealt with managing the capacity of their systems over the last few years.

Here are the results.

Has anything changed?

The first thing we asked was whether anything had changed with regard to system capacity. Has it been down, left pretty much the same, or scaled up – and if the latter, then by how much? Note that we talked about behaviour over three years, and asked those who’d not been running their systems for that long to dig out the crystal ball and tell us what they expected to happen over that period.

Of the 136 respondents, just one said they had noticeably reduced the capacity of their systems. And if we’re being honest that surprised us a little: cloud tech on a pay-as-you-go model screams out for occasional capacity reviews, and one might have expected a few more people to have spotted opportunities to consolidate or downsize where capacity exceeded demand. But the people have spoken, so maybe our expectation was misplaced.

A tad under two-fifths (38 per cent) said their capacity had stayed the same. No surprise here: if you’re using on-premises kit you should be buying stuff that will cater for a few years’ growth, and in the cloud you can get great value by committing to a given level of computing for an agreed period, which makes it affordable to provision for growth.

Then we have those who’ve grown their estates. About 22 per cent said they’d added around 50 per cent to their systems, and we reckon that a decent chunk of this will have happened since the spring of 2020 with Covid-induced dashes to implement VPN services, remote desktop services and accelerate moves to the cloud.

And a further 20 per cent or so said they’d doubled their capacity – which frankly means that even with the chaos of the pandemic they’ve clearly been adding systems and applications to their business-as-usual world alongside any quick fixes they’d done to make home working doable.

At the very top of the scale we had a non-trivial number – a pretty whopping 18.4 per cent, or nearly a fifth – who have increased capacity by 200 per cent or more over three years: this shows significant growth in companies that seem to have ramped up business and computing despite everything that’s been going on in the world for 18 months.

Do you test before you make the change?

This correspondent has an interest to declare here: when I started working in the Channel Islands back in 2007 it was as a performance tester on a multi-million-pound software deployment project. I wrote automation scripts and multi-threaded apps that would simulate hundreds of users hammering the SQL Server-based back-end database, and it led to significant changes being made to the underlying code to make it scale better.

And now, in 2021, 14 per cent of respondents to our survey told us that they did absolutely no performance testing prior to putting something new live.

Having picked our collective jaws up off the floor, we are reassured that about a third (31.6 per cent) said that they use “pre-defined performance metrics as part of the design and test plan, with electronic measurement of actual performance.”

A roughly similar number (33.1 per cent) use anecdotal, qualitative feedback from users as part of the test process – and we shouldn’t pooh-pooh such an approach because what really matters when all’s said and done is that the users and the customers they serve consider it “fast enough.”

Not with all the quantitative measures and five-decimal-place numbers in the world can you know that the people using the system think it’s dog-slow and start making their own workarounds. Oh, and a fifth of people used a combination of numbers and user testing feedback – which strikes us as a very sensible compromise.

How does it perform?

So, having done some testing (or not, if you’re in our sinning 14 per cent) pre-go-live, how do you monitor whether expectations have been met? A quarter basically said: wait for the users to moan: yes, 26.5 per cent said they react to users approaching them with feedback. A lower 17.6 per cent work on user feedback but go out proactively to the users to ask how the system is behaving and performing.

A tad under a quarter use electronic measures and benchmarks to establish if the system is doing what it’s supposed to at the speed it’s supposed to, and again we have a refreshing number – 31.6 per cent – who monitor electronically and use user feedback to get a more complete picture of what’s occurring.

In summary, then …

We reckon there were a few surprises in our survey snapshot. The biggest was, of course, the 14 per cent who said they didn’t do any performance testing before going live with something: while such testing can be expensive and time-consuming, doing none at all strikes us as a little odd.

Next was the fact that only one of 136 respondents said they’d scaled systems down: I’m sure some of you reading this will have been regaled by colleagues in the pub, telling you they just got a tasty bonus from rationalising their estate and saving a wad on cloud hosting or from decommissioning legacy kit.

And finally, we were thinking that there would be a few more out there who were using a hybrid model for pre- and post-go-live performance testing and analysis, by employing both human feedback and technical measurements.

But to see almost a fifth of people saying they’d added 200 per cent or more to their systems’ capacity was an eye-opener for us. It suggests that a lot of companies out there continue to grow their technology despite all the COVID-19 speed bumps that have appeared throughout 2020 and 2021.

Though of course many of these might be the 14 per cent who didn’t check what the performance of their new system was going to be like and have spent the last year and a half reacting to unsolicited user feedback and going Right-click -> Add storage … Right-click -> Add RAM. ®