Avoiding the trap: Proper estimation of storage needs

Over the years many of my clients have come to me with an annoying problem: the need to constantly buy ever-increasing amounts of storage. Every time they built a system based on previous growth numbers it turned out disastrously wrong. Storage systems have a nasty tendency to grow in punctuated equilibriums—following a normalized growth curve, then exploding at odd moments. Tracking these sudden explosions in storage needs back to their source allows us to more accurately estimate need, thereby smoothing out future capacity planning efforts.

Sources of storage needs
When I went over my compiled notes about these projects I realized something. My raw notes contained vast amounts of information about storage utilization, but we had compiled it all into a single growth percentage over time. This is, unfortunately, a classic error in data analysis: applying an aggregate number to a real-time sample. Sure, over time the two will match, but specific instances may well differ radically.

It is in the places where the two differed that my clients ran into trouble. For example, one client estimated 20 percent growth per year for four years based on the last decade of data. Unfortunately, two years into his plan, his business users demanded a 100 percent increase in existing storage. When they checked the short-term rather than long-term trends, he discovered the demand fell into reasonable predictions but he could not meet the requirement in time. I worked with him to deploy additional storage as quickly as possible while gathering data on what happened.

In this specific case, my client did not account for his company’s established six-year cycle of shifting in and out of ERP solutions. Every three to four years for the last thirty, they had installed a new system to run the company. The storage team inherited responsibility to maintain the development channels for the “old system” and run simultaneous development environments for the “new system,” effectively doubling their need for storage every time the cycle started backing up. [Continued…] [Shannon T. Kalvar]