I previously managed an online marketing application responsible for housing an unbelievable number of assets to generate high-resolution, production-ready PDFs for customers around the nation. And as most consumer-focused companies do, we maintained historical records so customers could easily reprint files from years ago.
Our IT manager was constantly showing me fancy charts and graphs of how quickly our data was growing and how I needed to come up with an archive strategy. The task was daunting – how was I to decide what was important and what was not? At the time, I felt this was a one-time, unique business problem and we just needed to throw a little capital at the issue and it would go away.
Looking back, I was completely wrong.
We faced the same business-problem occurring everywhere: the need to create and implement an archival strategy was not unique to our platform and it was not going away. Building an archival strategy to offload unreferenced data from your expensive SAN is an important problem and one that requires the attention of more than one individual. In fact, it impacts all of your business units. Just a quick list:
- Finance: Retention policies required for accounting purposes related to historical records
- Legal: Archival policy requirements for eDiscovery
- Marketing: Data required for historical reporting, including trending data for your customers
Businesses are constantly strategizing what could be offloaded from their expensive SAN system. If something wasn’t accessed for six months, is it okay to archive? And archival to us meant putting it on a disk that could be pulled from storage if the customer called and requested the information. If I selected the wrong files, there could be significant manpower required to make the files active.
Plus, the act of archiving was difficult to manage. Did this mean that I would have to archive an entire data set to be efficient? It was easy to make a decision for a file here and a file there, but when it was required to make a firm decision and determine an entire bucket of data – it was impossible for me to make that final decision.
Looking back, these are the things I wish I knew:
- It is best to not make archive strategy decisions when IT is waving the red flag. An incremental archive strategy is much easier to manage than an unplanned and disruptive data migration.
- It is much easier to migrate in blocks over the course of a few months than it is to determine which Midnight to 6 a.m. timeframe works for your customers’ SLAs to ensure no impact on the system.
- An archival strategy under pressure increases risk. There is risk in ensuring legal and compliance needs are met. There is risk in ensuring no impact to your end-consumers. There is risk in ensuring the proper data is being archived.
- Archiving the data is not the end of the line. Today, you could determine a significant amount of unreferenced data to offload, but without defining proper retention policies, you will one-day face a wall of disks that need to be cleared or approved for deeper archive.
- Removing unreferenced data from your SAN server is impactful. When a significant amount of data is removed from your SAN, decreases in your backup windows occur. For those managing to SLAs, this reduction in backup windows can be very impactful on your application strategy.
But most importantly, I wish I had known a decision on a budget does not mean data archival to disk or tape. In fact, a solution such as LightBridge ARCH, provides data archival that is scalable, reliable and cost-effective. Plus, you gain additional features:
- Secure data allowing authorized access as defined.
- Data remains searchable.
- Data is located in seconds, not hours fumbling through disks or tape.
- Data is accessed through script, not manpower.
Hopefully, your IT team will recognize and appreciate your newfound appreciation in listening to their concerns.