Containers and Microservices
Containers and microservices are helping enterprises achieve an automated flow of applications and data. “For applications, containers and microservices have standardized the distribution and delivery of code, which has helped simplify DevOps,” said Raheja. “The intersection of DevOps, containerization, and microservices has increased cloud adoption and accelerated the building of modern applications. For data, the configuration required can be treated as code, and the environments can be quickly spun up and down for data pipeline analysis and training. To that extent, containerization does help enterprises achieve some level of automated data flow.”
These approaches are important to increasing confidence. “Microservices break monolithic applications into decoupled, more easily understood pieces that decrease the automation effort by dramatically reducing the what-could-go-wrong surface area,” said Zweben. “Containerization helps automation in a similar way, but at the infrastructure level,” he said. In addition, containers abstract away the underlying infrastructure, allowing automation to bypass Byzantine platform configurations.
At the same time, underlying data needs to be a component of microservices and container environments. “The challenge for data management isn’t the containerization perspective, but the data itself,” Raheja cautioned. “Getting data tested in various environments is challenging, but discovering unknown data and cataloging it is an even bigger challenge.”
As with DevOps, many enterprises seek to adopt containers and micro-services, but execution is still poor. “For example, having the mindset of only going through APIs instead of databases creates a ‘just-get-it-done’ mentality, which then leads to rigid implementations that are hard to change and difficult to understand,” said Robert Castles, principal and chief technology officer for PMG. “Ideas and concepts like exposing back-end data to end users is challenging to users in a B2B context. For example, is the enterprise data searchable? Can it scale to end-user expectations for speed of queries? What are the performance standards of the target system? Is it reliable enough to count on, and how will unavailability impact the application?”
Advice on Automation
To achieve a more automated, data-driven environment, data managers need to think about the way data and applications are managed in profound ways. The following measures will help companies on their journey to automated, data-driven processes.
Move incrementally. “Each step in data analytics should be evaluated to determine how automation and intelligence could improve the process,” said Kohli. “In many enterprises, data managers should seek to implement DataOps for greater agility in their delivery of applications and data. When implementing new processes like DataOps, data managers must align relevant stakeholders to ensure streamlined operation across the enterprise—from engineers and analysts to data scientists and executives. Analysts and developers should also receive a steady stream of feedback so they can move quickly to identify and implement corrective measures.”
Decouple applications from data. “Work with application teams to, as much as possible, decouple changes to the data from changes to the application,” said Wallgren. “If you’re able to push data changes ahead of application changes, you can reduce risk
significantly. Of course, this doesn’t come free, but the effort pays off as you can often avoid having to roll back data changes if it’s necessary to roll back application changes. Organizations that have done the work to decouple their data from their applications are in a better position. For example, architecting an application so you can roll out a schema change before the application code that uses that change reduces quite a bit of the risk involved.”
Leverage the power of AI. “With the right AI in place, organizations can begin to automatically pinpoint the root cause of any performance problems and take precise steps to remediate them,” said Gandhi. “Not only does this level of intelligence help address issues of complexity, but it also lays a foundation upon which organizations can build an automated flow of applications and data across their DevOps toolchain, from continuous delivery pipelines to operations.”
Measure progress. Progress should be measured in terms of how much you and your team are finishing, said Kevin Kline, principal program manager at SentryOne. “Lots of managers measure how much activity is happening across their teams, but not how much is finished, delivered, and approved by the customers that they service,” explained Kline. “They are measuring how busy they are but not how productive they are. To achieve agility, many managers need to change how they are measuring the success of their teams. Once they make that change, other team improvements will quickly follow by detecting and remediating those issues that are holding them back from finishing projects that they’ve started.”
Encourage collaboration and teamwork. “The success of adopting automation or containerization depends on the tight working relationship between infrastructure and development professionals,” said Klang. “Neither team can unilaterally design a successful system. Teams should share common goals around availability and speed to delivery, and have a strong agreement about how resources (compute/network/storage) will be made available.” For data managers, the worst-case scenario is building a system that nobody wants to use. For developers, they lack the access or authority to deploy systems themselves, which can lead them to look externally for solutions, said Klang. “Working together is the only way for both sides to be successful.”
Reevaluate how teams are structured. “While IT, developer, and operations teams have traditionally worked in silos, and while these teams have often remained disconnected from the business, today’s IT complexity, coupled with customers’ demands for amazing digital experiences, requires an entirely new approach,” said Gandhi. “It’s critical to empower all teams to clearly see how their work is aligned with, and supports, other teams and—most importantly—how their work ties back to business goals.” This can be done by sharing insights on IT performance and customer experience that all teams can share, using a common language and common view into how user experience and performance are impacting business outcomes, Gandhi explained.
Boost data quality. “Going from on-premise solutions to cloud-based platforms, for example, changes neither the data utilized nor the data produced,” said Lee. “If you deliver garbage data to your cloud applications, you still have the same data quality problems as before—and, in the era of AI and machine learning—the consequences of poor data quality can be even more severe. That’s because there’s a multiplier effect with machine learning—first in the historical data used to train the predictive model, and second in the new data used by that model to make future decisions. Establishing stringent data quality standards and practices that are sustainable and can be adapted for the future is a crucial first step. Data managers need to efficiently cleanse, transform, and enrich data in a scalable manner for accurate analytics. This can be a challenge, especially for large sets of data. Enterprises should look to tools to make the process faster and more efficient.”
Apply culture change to ideas about data and data platforms. Once corporations have embraced the idea of smaller, cross-functional, agile teams, those teams need to be working from a common understanding of the business and their customers, Newton stated. “That common understanding is expressed in a common data platform. Containerization in particular and microservices in general are generating too much complex data to allow the old silos to exist. A shared data pipeline and data platform will allow enterprises to unleash their data-hungry teams and encourage collaboration.”
The greatest wins in automation “come from rethinking the problem being solved,” agreed Klang. Just as virtualization replaced metal servers, and containers are replacing virtualization, the organization and the people both have to be part of the change,
or it is almost certain to fail, Klang observed.