Thanks to an emerging generation of automation tools, there’s finally an opportunity to clear away the onerous and administrative tasks that have bogged down IT and data professionals for decades. The technologies that are freeing them up from onerous data management tasks are guided by robotic process automation (RPA), robotic data automation (RDA), low- and no-code platforms, and other forms of data automation that are making their way into the enterprise. It’s a matter of identifying productivity opportunities and applying the right tools at the right times.
The case for increased data automation is clear. “Data teams are spending significant amounts of time on service requests like infrastructure, user provisioning, and incident coordination and communication,” said Tina Huang, CTO and founder of Transposit. “Teams today are often manually creating tickets, Slack channels, and Zoom meetings, plus communicating with stakeholders. Data teams must ensure internal customers using data have access to the data they need and real-time updates about interferences with that data.” Other tasks ripe for automation include log parsing, correlation, permissions and access, and more.
However, while database automation and practices have been surging, this has not provided relief for data teams tending to day-to-day tasks. In a recent survey by Unisphere Research, a division of Information Today, Inc., 43% of data managers said the amount of resources spent on ongoing database management is severely limiting their competitiveness—an increase of 65% over a previous survey in 2020. It is notable as well that a total of 86% agreed that, to some degree, their administrative tasks are inhibiting corporate growth (“2022 Quest IOUG Database Priorities Survey”).
TIME SINKS
“Time is wasted on a day-to-day basis as teams work to improve application performance or respond to issues with applications,” said Cass Bishop, director at ISG Automation. In addition, there are other time sinks, such as “pulling search-based queries for their teams, and specifically frontline workers,” said Cindi Howson, chief data strategy officer at ThoughtSpot. “Frontline workers need the ability to answer data questions in the same way they’d search for information in their daily lives—which isn’t possible with cumbersome, legacy analytics tools that require them to constantly go back to their data teams.”
IT and data teams are also bogged down with “support requests for one-off API integrations between SaaS apps,” said Rich Waldron, co-founder and CEO at Tray.io. “IT is understaffed and overwhelmed due to greater support demands as pandemic-era distributed teams rely on an ever-increasing array of cloud apps. As more line-of-business teams use more tools that don’t talk to each other, they make increasing requests for integrations across those tools. The average IT team has a project backlog of 3-to-12 months. In the meantime, IT continues to face increased demands for strategic projects such as digital transformation and improved information security.”
With the rise of more comprehensive data analytics across the edge, the need to tend to tasks such as time-series aggregation is only growing as well. “The trend is clearly emphasizing the importance of understanding how and why the data changes over time, simplifying the mechanics of correlation and causation, and surfacing easy to spot trends,” said Francesco Crippa, vice president of platform engineering for Uniphore. “Who doesn’t like a chart where you see a single data point changing over time? But time-series data usually comes with the heavy need of aggregation. It’s painful to see the amount of small tasks in cleaning, aligning, normalizing, and adjusting all the differences in time-series aggregations when dealing with non-centralized data.”
Still, said Chris Wood, senior account executive at ASR Analytics, a GCOM Company, “IT professionals are typically ahead of the business side of an organization when it comes to automating processes. There is, however, a considerable amount of manual repetitive processes within the business side—such as working approvals for user requests that require manual verification, system cross-checking, and data validation.” This is where automated approaches such as robotic process automation make a difference, he added.
ENTER DATA AUTOMATION
Data automation offers the most compelling path to building an intelligent data supply chain. The value of data comes from its quality. With the mundane, time-consuming, and costly tasks associated with quality becoming automated, any enterprise can realize ROI a lot quicker, and time-to-market can be much faster. Approaches include robotic process automation, robotic data automation, DataOps, MLOps, OCR (optical character recognition), iPaaS, ETL/ELT, and more.
“Automation should start with IT, specifically, the help desk,” advised Pat Calhoun, CEO of Espressive. “A majority of remote employees’ top complaint with IT is waiting too long for their issue to be resolved. When an employee needs assistance from the help desk, they need it immediately—just like help from virtual assistants, like Amazon and Alexa, in their personal life.”
“For organizations looking to begin their automation journey, RPA is a great starting place,” said Wood. “When the right process is identified, RPA will rapidly generate an ROI. This means that a successful RPA deployment in one area of the business can encourage employees to identify other processes that could be automated. That can build momentum for growing the use of RPA and other modernized approaches, such as AI, machine learning, and natural-language processing, to improve operations and services.”
Software bots “leveraging RPA and iPaaS can help organizations optimize business workflows, leading to improved efficiency and productivity,” said Calhoun. “For instance, instead of requiring employees to traverse the complexity of an HCM [human capital management] tool to submit a vacation request, an organization may decide to build a worker bot to automate the submission of vacation requests. Frequently, the issue with these worker bots is discoverability. An organization can end up with numerous worker bots covering a multitude of functions, making the problem worse.”
Emerging data automation frontiers also involve data sharing and access provisioning, said Nitin Mathur, senior vice president of customer experience at Privacera. “Previously, policy enforcement and access provisioning were the domains of IT experts who resorted to database-specific admin tools and coding to create or modify policies that would grant access to a data set, or manually create some masking on sensitive data elements in the provisioned dataset. Today, they can be automated to a large degree and also federated to the business owners themselves.”
Look at low- and no-code automation technology to free up IT and data staffs from mundane, time-consuming tasks, Waldron urged. This technology “democratizes access to improving technical processes that might formerly have lived exclusively with IT.”
CHALLENGES
What are the challenges to fully implementing data automation approaches such as RPA, RDA, or other forms of automation? There may be times when these are not necessarily the best approaches, industry observers agreed. Data automation often leads to increased “enterprise IT sprawl and a lack of standardized monitoring tools,” Bishop cautioned.
“If you have seen the tech scene before the late ’90s and the way software was written, designed, and organized, a lot of RPA flows resemble that organization—only it’s spaghetti RPA instead of spaghetti code,” Crippa added. Consistency is also an issue, he said. “Imagine an RPA flow where you add a data point to two different systems: Is this data of one system more trustable than the other? Can the data in one system mutate without changing the data in the other system? With RPA and RDA, the job is much harder, and definitely not spontaneous.”
Data teams “diving straight into end-to-end automation soon realize it’s not scalable,” Huang noted. “Instead, they should implement solutions that automate incrementally, building trust and confidence in the technology. They should focus on smaller tasks that are repetitive and will provide immediate benefit—like ensuring observability of infrastructure, automating the creation of incidents, and instantly uniting the right team members.”
In addition, Huang continued, “another pitfall is when humans are left out of the loop. Automation should combine human judgment with machine logic. Teams often lack standardized processes that help them intelligently understand which automation to execute. While some workflows are suited for full automation, most processes require human judgment and team collaboration.”
RPA is rapidly becoming “one of the fundamental pillars of data automation,” said Crippa. “Unfortunately, RPA is a broad set of technologies and platforms that span in many directions. Not all the needs in automation are generally available universally in every RPA implementation. Choosing the RPA platform that fits the actual needs is not an easy task.” Crippa predicts that RPA will evolve organically and be distributed, tied to the specific automation needs of the processes or applications involved.
“If data automation initiatives are scattered across organizations, using separate approaches and technologies, and lacking centralized governance and oversight, organizations will be left with higher costs and lower impact,” said Param Kahlon, chief product officer for UiPath. “Managers will need to make crucial decisions, such as which automation technology to standardize on, which organizational capabilities will need to be built out, and how to ensure good governance, security, and quality.”
With both RPA and RDA, “many companies make the mistake of taking these approaches to try and create orchestration layers for end-to-end processes, and they find that the technologies do not scale to meet their needs,” said Carter Busse, CIO of Workato. “While analyst labels such as ‘hyperautomation’ or ‘intelligent automation’ are used to describe the collection of tools that companies try to piece together to achieve their end-to-end goals, different teams and tools can lead to fragmentation across the company.
IMPLEMENTING DATA AUTOMATION
The most important step in introducing automation into a data strategy is to commit to a data governance initiative, said Sharad Varshney, CEO of OvalEdge. “Data governance is needed to streamline data management processes and incorporate automation via a data catalog, which will centralize and automate data governance programs.”
As the effort gets underway, Wood advised it may be necessary to perform “an environmental scan of the business to understand their mission and identify where manual processes that take up the most staff time exist. Typically, these processes will also have backlogs that are aging due to not enough staff time available to process.”
The simplest form of RPA “replaces the rules-based, repetitive tasks that a user would perform on their computer with an automated set of processing steps,” Wood added.
Focus on enterprisewide automation, “which ensures a single orchestration layer for integration and automation without the need for a collection of disparate tools handling different functions,” Busse urged.
BENEFITS
Still, there are already numerous tangible benefits data automation is delivering to businesses. For starters, it enables self-service analytics, which can reduce IT and data management teams’ workloads up to 80%, Howson estimated.
This is enabled through companies replacing manual tasks with software robots that manage data, handling upfront data preparation tasks that often took days and weeks to accomplish. As a result, data automation “delivers an ROI that other technologies struggle to match,” said Kahlon.
Greater automation of the ETL pro-cesses—and increased quality and reliability of the data being ingested—ultimately saves the business valuable user time that it can then spend on higher level tasks, said Wood. “By saving staff time, automation also helps accelerate service delivery, save costs, and help avoid errors—meaning better outcomes and experiences for the customer.”
Multiple data points are transformed and aggregated into meaningful information, said Crippa. “It’s easy to collect millions of data points, but it’s hard to present those in a way that allows clear and simple decision making based on that. Data automation allows for a methodical approach to data transformation in continuous iterations. Simple iterations that can be done week after week without the need of intensive upfront planning and design.” And, with easier to read and more accurate data that is always available and always up-to-date, business decisions are always more effective, proactive, and accurate, Crippa noted.
Data automation “can optimize organizations’ overall operations, leading to higher production rates and increased productivity,” Calhoun agreed. “Automating IT questions and tasks can empower employees, as they are spending less time on hold, waiting for help-desk agent assistance. In turn, IT teams are less overwhelmed and burdened, so they can help where they are needed most, increasing their productivity.”
In addition, the shift to self-service through low- and no-code technologies provides “IT teams an opportunity to enable faster innovation among lines of business,” said Waldron. “Low code gives non-technical users the power to automate complex processes that not only increase productivity but remove the limitations of their current software tools. LOB [line of business] can then rapidly scale operations and pave a path for significant future growth. IT teams have a golden opportunity to take the lead and empower LOB users to self-service day-to-day challenges while thinning out their own project backlogs and freeing up much-needed bandwidth to focus on higher-level projects.”
There are three ways data automation delivers new capabilities to enterprises, Busse explained. First is democratization, in which people outside of IT are empowered to build with low- and no-code approaches, while keeping guardrails in place to ensure they can automate safely. Second is greater orchestration, as automated data platforms and tools “take an API-led approach to connecting systems and orchestrating data, processes, and user experiences,” Busse said. Finally, data automation results in a process mindset, in which “instead of focusing on tasks, the focus is on processes first and foremost for more impactful outcomes.”
Data automation “enables businesses to pull value and insights from their data at a much faster rate, which in turn facilitates higher productivity,” said Mathur. “It also allows for better business satisfaction with the data and analytical processes as teams are now able to self-service data governance—not just analytics. Automation can result in better compliance and integrity of security and privacy programs and also simplifies processes as it allows you to greatly reduce the number of control planes for data and the number of tools in use.”
Data automation approaches such as RPA “are very useful in cases where the standard operating procedure is well-?defined and doesn’t change,” said Vipul Valamjee, senior engineering leader at Altimetrik. “Any digital organization needs to align to the evolving business environment,” and this is one reason why RPA and other data automation processes represent a journey and not a destination, Valamjee noted. “A constant effort is required to keep that automation up to the current requirements of the business.”