Microsoft is on the march with multiple upgrades and improvements across the span of its cloud data products. Here are a few of the more prominent highlights.
Azure Elastic Database Jobs Now in Public Preview
If you’ve been waiting to move your SQL Server workloads into the Azure cloud, it is time to celebrate. Microsoft is making Azure SQL Databases one very big step closer to feature parity with on-premise SQL Server. The new release of Azure Elastic Database Jobs is a significant upgrade. Elastic Database Jobs is your Azure replacement for SQL Agent, enabling you to schedule and run T-SQL scripts across a large number of databases in parallel. Elastic Database Jobs is a service that supports any combination of database targets, including all the databases on a server, in an elastic pool, singleton databases, and across multiple servers, multiple pools, and different subscriptions, with the added ability to exclude any that you choose.
Unlike the earlier release, which was customer-hosted, this release is a fully integrated Azure service requiring no additional services or components. It also adds many features to ease automation of T-SQL jobs using PowerShell, REST, or T-SQL APIs against your target databases. You can use it to do all of the usual preventative maintenance, index rebuilds, and report processes, just as you would use SQL Agent for on-prem SQL Servers. You can also configure a limit to the number of databases a job might run against in parallel to ensure optimal use of resources or to avoid bumping up against limits.
You can learn more about Azure Elastic Database Jobs at https://docs.microsoft.com/en-us/azure/sql-database/elastic-jobs-overview or you can check out the specific Azure blog post sharing all the details of this latest launch at https://azure.microsoft.com/en-us/blog/azure-elastic-database-jobs-is-now-in-public-preview.
New Azure Data Factory Capabilities
Azure offers a variety of service-level agreements (SLA) at different fee structures and in various degrees of completion, such as private preview versus public preview versus general availability (GA). So it’s sort of a big deal when a specific feature set is released to GA. That is the case with Azure Data Factory (ADF), which now falls under the GA SLA.
ADF V2 features a new browser-based UI located in the Microsoft Azure Portal that is easier and better designed than past versions, enabling you to quickly get data pipelines into production with its slick drag-and-drop interface. You can use ADF to build, schedule, and administrate your extract-transform-load (ETL) processes at scale wherever your data might live, whether on-prem or in the cloud, all while shielded with enterprise-quality security features. The new release allows existing SQL Server Integration Services (SSIS) users to “lift and shift” existing SSIS packages into the cloud, and run SSIS as a service.
The release also provides connections to more than 70 data sources. You can build ETL pipelines in the cloud with ADF that transform data at scale using Spark with HDInsight on-demand clusters. You can also do that with Azure Databricks Notebooks. Further application development flexibility is provided via SDK support for Python, .NET, REST, and PowerShell. In addition, ADF now supports the ability to trigger the activation of a data pipeline based on the occurrence of a given event. For example, you might want to trigger a pipeline any time a file is uploaded to your Azure Storage account. Details are at https://azure.microsoft.com/en-us/blog/event-trigger-based-scheduling-in-azure-data-factory.
More information is available at https://azure.microsoft.com/en-us/blog/azure-data-factory-new-capabilities-are-now-generally-available. And, check at the bottom of the article for many more links to hands-on labs, tutorials, ADF ebooks, and white papers.
Online Resumable Indexes Creation
Azure SQL Database, Microsoft’s flagship relational database product in the Azure Cloud, always has new features in the works. Here’s a feature now in public preview which I really like—online resumable index creation. Online index creation means that you don’t have to kick off any users whenever you need to create an index. Plus, it uses very little log space, banishing a long-standing problem for SQL Server and Azure SQL DB where customers run out of log space during index creation and index rebuilds, thus forcing database outages. As you can gather from its name, it enables you to resume an index creation task from where you left off (due to any number of reasons, such as a failover, database disconnection, manually pausing the task, system crash, etc.) rather than restarting from the beginning. You can learn more about this feature at https://azure.microsoft.com/en-us/blog/resumable-online-index-create-is-in-public-preview-for-azure-sql-db, as well as see it in action in this YouTube video at www.youtube.com/watch?v=mMRpUEt tUQY&feature=youtu.be.