The amount of data managed by the world’s enterprises is growing. According to one source, the total amount of data created, captured, copied and consumed globally was about 64.2 zettabytes in 2020 — equal to a trillion gigabytes. Unsurprisingly, companies report that the cost of storing their data is also climbing. In a 2018 Enterprise Storage Forum survey, business leaders said that the high costs of operation, a lack of storage capacity, and aging equipment were among their top concerns.

The rising costs of storage have pushed many companies to adopt cloud options, which offer the advantage of low entry costs. But with costs inching up as more businesses move online — a Pepperdata report found that more than one-third of companies have cloud service budget overruns of up to 40% — IT leaders are exploring alternatives.

On the cloud side, a nascent crop of startups are applying AI to the problem of managing cloud spend. Vendors like Densify and Cast AI claim that their AI-powered platforms can recommend the best storage configuration for a companies’ workloads by taking into various requirements. Other technology providers have turned their attention to on-premises systems, creating algorithms that they claim can reduce storage costs either with hardware suggestions or novel file compression techniques.

“Data storage today suffers from several challenges: Storage deployments are often made up of a variety of different storage media such as memory, flash, disk drives and tapes. In addition, organizations run multiple storage arrays based on access protocols … or based on criticality of the workloads,” Gartner research VP Arun Chandrasekaran told VentureBeat via email. “The usage of AI has the potential to streamline data lifecycle management based on criticality, performance, security and costs requirements of data.”

Cloud optimization

During the pandemic, the pressure to digitize operations led a record number of companies to move to the cloud. According to a recent survey from O’Reilly, 90% of organizations were using cloud computing of some kind in 2021, while Flexera’s State of the Cloud Report shows that 35% of companies spent more than $12 million on cloud operations in 2021.

The adoption trend gave rise to startups developing AI-powered platforms designed to adjust usage to reign in expenditures. One is Densify, which analyzes workloads across private data centers, Amazon Web Services, Microsoft Azure, Google Cloud Platform and IBM’s cloud offerings to determine how much CPU, RAM and storage they need — then suggests ways to save. Densify can use already-available log data to begin optimizing right away. After that, the platform will continue to review cloud providers’ pricing changes, applications’ needs and new products to find where customers can reduce expenses further.

“Usually within two to four weeks, you’ve got 50% of the savings,” CEO Gerry Smith told VentureBeat in a previous interview. “Depending on where the savings are, within another two to four months, [you’ll get] 100% of the savings.”

Cast AI, a Densify competitor, similarly leverages AI to optimize cloud spend. Supporting major cloud service providers, the platform connects to existing clouds and generates a report to identify cost-saving opportunities.

“We have other models that use global datasets for market characteristic predictions,” CEO Yuri Frayman told VentureBeat in October 2021. “For example, we train a global model to predict instance preemptions by machine type, region, availability zone and seasonality. This model is shared autonomously across all customers, and all the data is used to retrain the model continuously.”

On-premises and compression

For companies that haven’t made the move to the cloud — or who have their data spread across cloud and on-premises environments — there are solutions like Accenture’s Storage Optimization Analytics, which combines search and AI to understand enterprise content and automate data classification.

Accenture claims that it reduces storage costs by detecting duplicate or near-duplicate content, helping customers move or archive the right data at the right time. Storage Optimization Analytics also automates migration to lower-cost storage and tracks storage savings, computing the overall return on investment (ROI).

IT provider Rahi Systems offers a similar service called Pure1 Meta, which uses AI models to predict capacity and performance and provide advice on workload deployment and optimization. Pure1 Meta can run simulations for specific workloads, generating answers to capacity planning questions while ostensibly helping to increase resource utilization.

AI is also increasingly playing a role in file compression. For videos, music, and images, AI-based compression can provide the same — or close to the same — level of visual quality with fewer bits. Another benefit is that it’s easier to upgrade, standardize, and deploy new AI codecs versus standard codecs, since the models can be trained in a relatively short amount of time and — importantly — don’t require special-purpose hardware.

Websites like Compression.ai and VanceAI leverage models to compress images without compromising on quality or resolution. Qualcomm and Google have experimented with AI-driven codecs for both audio and video. And Alphabet-owned DeepMind has created an AI system to compress videos on YouTube, reducing the average amount of data that YouTube needs to stream to users by 4% without a noticeable loss in video quality.

Looking to the future

Gartner’s Chandrasekaran notes that the adoption of AI technologies for data management, which fall under the category of “AIops,” remains quite low. (AIops platforms aim to enhance IT by leveraging AI to analyze data in an organization’s from tools and devices). But he adds that the pandemic has been a catalyst for adoption as organizations strive to automate faster to respond to “rapidly changing” circumstances.

Recent surveys agree. According to Emergn, 87% of companies expect their investments in automation skills to increase over the next 12 to 26 months. And in a 2020 K2 poll, 92% of business leaders said that they consider process automation vital to success in the modern workplace.

“There is a lot of ‘AI washing’ in the industry today. Hence, vetting vendor claims and deploying a solution that delivers ROI can be frustrating. AIops requires a lot of integration,” Chandrasekaran said. “For teams that aren’t skilled in architecting and maintaining complex data environments, a robust AIops deployment may become a pipe dream. There also needs to be a cultural change, where organizations are willing to make data-driven decisions.”

Looking ahead, Chandrasekaran expects to see more “versatile” AI-powered storage management solutions beyond the products already on the market. These solutions could enable greater intelligent automation and remediation workflows through the use of AI, he believes.

“AI techniques can help optimize placement of data on the right storage tiers — balancing performance and costs. In addition, AI can help with better availability of data infrastructure, enabling businesses to access data faster and create a reliable infrastructure,” Chandrasekaran added.