Central Vermont Public Services expands use of Surveillance DB for critical database monitoring.
When you think of mission-critical services, perhaps none is as critical as electrical service. Not much can happen in modern businesses, government offices, or even homes without it.
Central Vermont Public Service (CVPS) is the largest electric company in Vermont. More than 159,000 customers in 163 communities rely on the electrical service CVPS provides. And, according to J.D. Power and Associates, a global marketing and surveying company, for overall customer satisfaction, CVPS continues to rank in the top tier of utilities in the eastern region, more than 50 points above the regional average.
To support that level of exceptional service, in turn, CVPS must have excellent IT support at its core. A Bradmark customer since the mid-1990s, CVPS first relied on Bradmark Technologies' Surveillance DB for real-time database monitoring and troubleshooting capabilities of its Sybase ASE databases and recently expanded its use of the product in its diverse IT environment. Surveillance DB is a proactive database solution that identifies problems before they occur and alerts administrators before they impact availability.
Multiplatform Environment
"We are a multiplatform department, supporting a variety of both custom, in-house developed and third-party applications," Wendy Perry, lead DBA at CVPS, tells DBTA. Perry knows CVPS' systems well. She has been with the company for 18 years, starting her career there as a college intern and joining as a programmer upon college graduation. She subsequently rose to the position of junior DBA before assuming her current role. Describing the breadth of the CVPS IT environment, Perry says the company's client server databases include SQL Server, Sybase and Oracle running on varying platforms including Windows, SUSE Linux, Red Hat Linux, and IBM AIX. "We also support some mainframe databases and applications."
Perry has been using Bradmark technologies for almost her entire career. "I have not seen any tool better or even one that comes close to the real-time monitoring capabilities of Surveillance," she says.
In fact, CVPS recently expanded its use of Surveillance DB beyond just Sybase ASE after deciding to upgrade its Surveillance software to the latest version. In the process, it was necessary to also upgrade CVPS' licensing agreements with Bradmark and, Perry says, CVPS actually found that it became affordable to expand Surveillance DB to more of its systems. CVPS now monitors all of its Sybase ASE, Oracle (excluding SAP Oracle systems), Sybase Replication Server, and Oracle RAC systems using Bradmark's Surveillance DB.
Expanded Use
"Since expanding Bradmark to include Oracle, I have found it much easier to monitor Oracle systems. Other ‘tools' I have used have not been able to provide real-time diagnostics," she says. "We have been able to quickly understand performance issues and resolve them in order to keep pace with critical project timelines."
As an example of the types of fundamental systems CVPS monitors with Surveillance, Perry points to CVPS' work management system, which runs on Sybase ASE. "The work management application, which is used to control the distribution line work business process, is critical to the company and the responsibility we have to serve the electric customer needs related to new and changed electrical service," says Perry.
All work to be performed in the field by CVPS line workers is designed using the work management application. The designs, which contain the bill of materials for the work, including GIS schematics, are provided to the line workers on work tickets, which they use to perform both repairs and initiate new service in the field. In addition, customer call reps also use the system, she adds.
This work management system is integrated with an Oracle system via Replication Server, says Perry, explaining that the company has a significant amount of custom database code that is executed by the replication system to properly manage the relationship between these two systems. Recalling one particularly difficult problem which Surveillance helped to resolve, Perry cites a situation in which there were severe performance issues in the Sybase (work management) database when transactions were replicating back from the Oracle database.
Problems Avoided
"At times, we had blocking for 30 minutes or more. This was severely impacting the usability of the work management application as a whole," Perry says. Many users were sitting in front of screens looking at an hourglass, while replication transactions also continued to get backed up waiting for things to clear. "We used Surveillance to see what was going on while the blocking was occurring. We could see that the blocks were caused by table scans on large tables. We could further see the query plan as well as the exact line of SQL code that was being executed. We realized that the issue was a problem with the code," says Perry. When the problem was identified and resolved, the database blocks stopped, speeding up the system as a whole.
The newest system CVPS is putting in is an automated meter data management system. "Once we get new meters installed, we will be getting interval data coming in throughout the day from all of the meters all over the territory, and so performance in that system is also critical," explains Perry. The new meters will send information at specified intervals through CVPS' AMI (automated meter infrastructure) system which will then feed into the meter data management system, consisting of two Oracle databases.
Recently, as part of the process of setting up and testing this new meter data management system, Perry says she and her team encountered a performance issue in the new system, causing the processing to take much more time than it should. They were not yet using Surveillance for this system, and after spending weeks trying to solve the problem, they still could not pinpoint the source.
"We did not have any real-time monitoring tools that could specifically and accurately identify the issue. I did not feel like I had a good ‘window' into the system to see what was going on." The team had many discussions around snapshot report findings and configuration parameters, but they were not getting helpful diagnostic information in a timely manner, says Perry. "The tools would make assumptions and give us reports about what they thought the problem was."
Since they had not yet made the licensing upgrade to include Oracle, they installed Surveillance in evaluation mode and saw the problem in minutes. "We were watching the processing and immediately we could see that the process was showing up as a long operation," Perry says. They could see the SQL that was running, and could see that it was a 10-row table that was spanning over hundreds of megabytes of space. "Because we had that right in front of us we could see exactly what it was doing, and it was giving us more detail in terms of where else to look."
By contrast, for example, says Perry, using the RAC sessions monitor in conjunction with the RAC long-running operations monitor, the team had previously been given the recommendation that an index be created. "An index on a 10-row table would not have resolved this performance problem," states Perry.
"It was in using Surveillance that we saw the long op, the scan and the size of the table, which led us to the underlying problem," she explains. "I run Surveillance using a one-second refresh interval so it is as close to real-time as you can get. Once the session popped up in the long op window, we could drill in for further detail such as SQL text and query plan." They immediately made changes and, says Perry, "it made a big difference." CVPS subsequently purchased the Surveillance DB licenses for that system as well, says Perry.
CVPS also utilizes Microsoft SQL Server database instances for its back-office-type applications, and although Surveillance is not used for those databases now, Perry notes that it has been considered because performance issues come up there also.
A Single, Efficient Solution
Having Surveillance DB for the Sybase and Oracle environment for which she is directly responsible is valuable, says Perry. "It makes things very convenient, having monitoring capabilities for everything I am responsible for from one interface, and not having to go to different tools or interfaces for different types of monitoring," she says.
And on top of the efficiency of the products, "the Bradmark team has been has been great to work with," Perry notes. "It seems that I have been dealing with the same technical support people at Bradmark for many years. They are very knowledgeable about the product, very helpful in resolving issues, and the response has been very timely."