Multiple ways to access your On-premise data in Microsoft 365 and Azure

If your organization is using a hybrid cloud environment, this post will shed some light to integrate on-premise resources with Microsoft 365 & Azure services. Hybrid integration platforms allows enterprises to better integrate services and applications in hybrid environments (on-premise and cloud). In this blog post, I will write about the different services & tools available with in Microsoft Cloud which allows you to connect or expose your On-premises data or application in Office 365. There are still many enterprise organizations on Hybrid mode due to various factors. It can be a challenging task to integrate your on-premises network but with right tools & services in Office 365 & Azure it can be easier. Find below the high-level overview & some references on how to

Access your on-premise data in Power Platform & Azure Apps (Logic Apps, Analysis Services & Azure Data factory)
Programmatically access your on-premise resources in your Azure Function app
Access on-premise resources in Azure automation account
Expose your on-premise Application or an existing WEB API in Office 365 cloud

Access on-premise data in Power Platform & Azure Apps (Logic Apps, Analysis Services & Azure Data factory):

The on-premises data gateway allows you to connect to your on-premises data (data that isn’t in the cloud) with several Microsoft cloud services like Power BI, Power Apps, Power Automate, Azure Analysis Services, and Azure Logic Apps. A single gateway can be used to connect multiple on premise applications with different Office 365 applications at the same time.

At the time of writing, with a gateway you can connect to the following on-premises data over these connections:

SharePoint
SQL Server
Oracle
Informix
Filesystem
DB2

To install a gateway, follow the steps outlined in MS documentation Install an on-premises data gateway. Install the gateway in standard mode because the on-premises data gateway (personal mode) is available only for Power BI.

Once the data gateway is installed & configured its ready to be used in the Power platform applications.

The other catch the gateway is not available for the users with Power Automate/Apps use rights within Office 365 licenses as per the Licensing overview documentation for the Power Platform. Data gateways can be managed from the Power Platform Admin center.

Shane Young has recorded some excellent videos on this topic for PowerApps & PowerBI.

To use in

create a Data Gateway resource in Azure.

High Availability data gateway setup:

You can use data gateway clusters (multiple gateway installations) using the standard mode of installation to setup a high availability environment, to avoid single points of failure and to load balance traffic across gateways in the group.

No need to worry about the security of the date since all the data which travels through the gateway is encrypted.

Data gateway architecture:

Find below the architecture diagram from Microsoft on how the gateway works

I recommend you to go through On-premises data gateway FAQ.

Integration Service Environment:

As per the definition from Microsoft an integration service environment is a fully isolated and dedicated environment for all enterprise-scale integration needs. When you create a new integration service environment, it’s injected into your Azure Virtual Network allowing you to deploy Logic Apps as a service in your VNET. The private instance uses dedicated resources such as storage and runs separately from the public global Logic Apps service. Once this logic apps instance is deployed on to your Azure VNET, you can access your On-premise data resources in the private instance of your Logic Apps using

HTTP action
ISE-labeled connector for that system
Custom connector

For the pricing of ISE, refer this link.

Programmatically access your on-premise resources in your Azure Function app

As you all know Azure Functions helps in building functions in the cloud using serverless architecture with the consumption-based plan. This model lets the developer focus on the functionality rather than on infrastructure provisioning and maintenance. Okay let’s not more talk about what a Function app can do but let us see on how to connect to your on-premise resources (SQL, Biztalk etc) within your function.

During the creation of a Function app in Azure, you can choose the hosting plan type to be

Consumption (Serverless)
Premium
App Service plan

Consumption based plan is not supported for the on-premise integration so while creating the app the hosting plan has to either premium or app service based plan & the Operating system has to to be Windows. On-premise resources can be accessed using

Hybrid Connections
VNet Integration

Hybrid Connections:

Hybrid Connections can be used to access application resources in private networks which can be on-premise. Once the Function app resource is created in Azure, go to Networking section of the App service to setup & configure. Go through the documentation from Microsoft for the detailed instructions to set this up.

How it works:

The Azure Hybrid Connection represents a connection between Azure App Service and TCP endpoint (host and port) of an on-premise system. On the diagram below Azure Service Bus Relay receives two encrypted outbound connections. One from the side of Azure App Service (Web App in our case) and another from the Hybrid Connection Manager (HCM). HCM is a program that must be installed on your on-premise system. It takes care of the integrations between the on-premise service (SQL in this case) with Azure Service Bus Relay.

Once the setup is done, you can create a connection string in Appsettings.json file or from Azure function app interface of your function app. After this you can access the data in your function app code.

I’ve found a couple of interesting blogs about this setup.

Azure Functions – Azure Functions Using Hybrid Connections by Stefan Roth
Using Azure Hybrid Connections by Peter

VNet Integration:

In the Networking features of the App service, you can add an existing VNET. An Azure Virtual Network (VNet) is a representation of your own network (private) in the cloud. It is a logical isolation of the Azure cloud dedicated to your subscription.

In Azure Vnet you can connect an on-premise network to a Microsoft VNet, this has been documented from Microsoft here. Once there is integration between your Azure Vnet & on-premise network and the VNet is setup on your function app you are set to access on-premise resources in your function app.

Access on-premise resources in Azure automation account:

Azure Automation is a service in Azure that allows you to automate your Azure management tasks and to orchestrate actions across external systems from right within Azure. Hybrid runbook worker feature allows you to access on-premise resources easily. The following diagram from Microsoft explains on how this feature works

I’ve written a blogpost recently about this feature for automating on-premise active directory.

Expose your on-premise Application or an existing WEB API in Office 365 cloud:

Azure Active Directory’s Application Proxy provides secure remote access to on-premises web applications (SharePoint, intranet website etc). Besides secure remote access, you have the option of configuring single sign-on. It allows the users to access on-premise applications the same way they access M365 applications like SharePoint Online, PowerApp, Outlook etc. To use Azure AD Application Proxy, you must have an Azure AD Premium P1 or P2 license.

How it works:

The following diagram from Microsoft documentation shows how Azure AD and Application Proxy works

Find below documentations on how to

Once the connector service is installed from your Azure AD application proxy, you can add an on-premise app as shown below

The above step will register an application with App registrations.

Summary: I’ve given some overview about the different services & tools to connect & integrate on-premise resources with Microsoft cloud. Hope you like this post & find it useful. Let me know any feedback or comments on the comment section below

Copy & Apply Site Template to a SharePoint site using Power Automate

If you have a requirement to copy a site template (Site Pages including images & webpart, site column, site content type, navigation etc) from an existing SharePoint site & apply it to a recently created SharePoint site, this blog post would be helpful.

Pre-requisites:

SharePoint site collection administrator
- SharePoint site with a custom list associated to a Flow
Access to Premium connector (Azure Automation) in Power Automate
Azure subscription to create Azure Automation Runbook
- PnP Power shell

SharePoint Patterns and Practices (PnP) community has developed a library of PowerShell commands (PnP PowerShell) that allows you to perform complex provisioning and artefact management actions towards SharePoint. On this example I will be using PnPProvisioningTemplate cmdlet’s to copy the pages including the assets & webparts to another site but you can do much more than this. Find the PnP cmdlets I will using

To generate a .pnp package (Site Template) from the source site

Get-PnPSiteTemplate -out template.pnp -Handlers PageContents -IncludeAllClientSidePages -PersistBrandingFiles

The parameter -PersistBrandingFiles saves all the asset files including the image files from the Site Assets library that makes up the composed look of page. Parameter -Handlers <Handlers> processes only the information passed to it. On the above example it processes only the Pages & its associated contents & not lists etc The PnP cmdlet Get-PnPSiteTemplate creates a package with extension .pnp which can be converted to a ZIP package by changing the extension to .ZIP from .pnp. Look at the Get-PnPSiteTemplate documentation for the various parameters it supports.

Once the package .pnp file is ready, the package can be applied to another site using the command Invoke-PnPSiteTemplate

To Apply the Template to a destination site (Apply template to site):

Invoke-PnPSiteTemplate .\template.pnp

If you want to test these commands in PowerShell console on your local computer, install the PnP module

Keep in mind before executing the PnPSiteTemplate commands, the site context must be created for both source & target site by creating a connection as shown below

Connect-PnPOnline -url "sourcesiteurl"
Get-PnPSiteTemplate -out template.pnp -Handlers PageContents -IncludeAllClientSidePages -PersistBrandingFiles
Connect-PnPOnline -url "targetsiteurl"
Invoke-PnPSiteTemplate .\template.pnp

For interactive login, use command

Connect-PnPOnline -url “sourcesiteurl” -UseWebLogin

Connect-PnPOnline -url “sourcesiteurl” -Interactive

Setup SharePoint List:

Till now you would have got some ideas about the PnP commands we will be using on the Azure Automation runbook, let’s now create the SharePoint list to collect the Source (Template to be copied from) & target URL (Template to be applied) for the SharePoint site. Find the list Schema for the List to be named as Site Template

Azure Automation Runbook:

The list is ready, let us now create the Azure automation runbook. I’ve written a post Execute SharePoint Online PowerShell scripts using Power Automate, it will help you with steps (Step 1 – Create automation account, Step 2 – Import SharePointPnPPowerShell Online PowerShell Module & Step 3 – Add user credentials) to create the automation account & runbook to execute the PnP PowerShell command for copying & applying the site template.

Step 4: Now we are good to create the Runbook, to create it click Runbooks under the section Process Automation and then click Create a runbook. Enter the Name of the Runbook ApplySiteTemplate, select the Runbook type to PowerShell and click Create.

Create Runbook in Azure Automation Account

Now let’s add the code by editing the runbook. The section Dynamic Parameters on the code will be passed from the flow. To connect to SharePoint Online site, we are using the SPO admin credentials created in Step 3. Find the code below

# Dynamic Parameters, will be passed from Flow
param(
  [parameter(Mandatory=$true)]
  [string]$SiteTemplateURL = "https://mydevashiq.sharepoint.com/sites/contosoportal",
  [parameter(Mandatory=$true)]
  [string]$ApplyTemplatetoURL = "https://mydevashiq.sharepoint.com/sites/contosositeportal"
)
# Credentials
$myCred = Get-AutomationPSCredential -Name "SPOAdminCred" 
# Connect to source site for creating the package or site template
Connect-PnPOnline -url $SiteTemplateURL -Credentials  $myCred
Get-PnPSiteTemplate -out template.pnp -Handlers PageContents -IncludeAllClientSidePages -PersistBrandingFiles
# Connect to destination site for applying the package or site template
Connect-PnPOnline -url $ApplyTemplatetoURL -Credentials  $myCred 
Invoke-PnPSiteTemplate .\template.pnp

The runbook is now created, you can test the script by clicking on Test Pane & pass parameters (Site URL etc) to test it. Click Publish button as shown below to publish so that it can be called from Power Automate.

You can also create the template (PnP Package) for a site & store it on a SP library. The PnP command to get the file

Connect-PnPOnline -url "siteurlwhichhasthePnPpackagefile" -Credentials  $myCred
Get-PnPFile -Url "/sites/sitenamewithPnPPackagefile/Shared Documents/template.pnp" -Filename "template.pnp" -AsFile

You can use a certificate instead of Username and Password to get a APP only access token to call PnP commandlet, for more details check the documentation for Connect-PnPOnline. It’s now time to create the flow to call the Runbook.

Power automate flow to call the Run Book:

You can now create a flow with automated trigger “When an item is created” from the SharePoint list created earlier to pass the Site Template URL & Apply to Site Template URL. Once the flow is created, add the action “Create Job” under the connector “Azure Automation” which is a premium connector.

Select the Azure Subscription which has the Automation account resource with runbook>Select Resource Group>Select Automation Account>Select the Runbook name which has the PS script. If there is a need to wait until the automation job completes then select Yes on the field “Wait for Job”. Enter the URL for SiteTemplateURL & ApplyTemplatetoURL

The flow is ready, run it to test now with parameters. I’ve used this sample to test a site (Template) which has

Customized home page with couple of standard webpart & images
2 more pages with images & other standard webparts

has copied to another site. If there is a custom webpart on the source site which is added to a page, make sure to deploy it on the destination site.

Summary: Take a look at the SharePoint starter kit PnP package to explore more about the usage of different features in PnP provisioning. This example can also be extended with Site design & Site script which has the capability to call a flow. Hope you have enjoyed reading this post and find it useful. If you have any comments or feedback, please provide it on the comments section below.

Automate the provision of On-Premise AD Account – Part 2

This post is in continuation to my previous post Automate the provision of Azure AD Account & License assignment – Part 1 for creating account in Azure active directory using Power Automate. On this post I will highlight the feature available in Azure Automation account which can be leveraged to create an On-premise AD account. Refer to this post for the usage of Azure automation account to interact with SharePoint online in Microsoft 365 using Power Automate.

Azure Automation is so easy to setup for automating tasks that interacts with

Azure (Azure AD, SQL etc)
M365 services (SharePoint etc).

Automation runbooks in Azure might not have access to resources in other clouds or in your on-premises environment because they run on the Azure cloud platform. To access local resources like On-premise Active Directory which lives behind the firewall, there is a feature within Azure Automation called as Hybrid runbook worker. Azure Automation Hybrid Workers extends Azure Automation into your private networks and allows running runbooks that interacts with resources such as on-premises Active Directory, SharePoint etc.

Hybrid Runbook Worker feature to run runbooks directly on the

Computer in On-premise network
Any secured network like a virtual machine in Azure behind the firewall
Cloud services like AWS etc

that’s hosting the role and against resources in the environment to manage local resources. Refer to this documentation about Hybrid runbook worker for more information.

The following image from Microsoft documentation illustrates this functionality:

Pre-requisite:

Azure Subscription to create
- Automation Account
- Log Analytics Workspace
Server (Windows or Linux based)
- Internet access
- Port: Only TCP 443 required for outbound internet access

The deployment instructions from Microsoft for

I’ve found a really interesting video on Youtube from Travis Robert regarding this topic to set this up on Windows workstation.

Once the Hybrid runbook worker is setup, you can write PowerShell script to create account in On premise Active directory. Find the link to the script to add a user to Active directory in Onpremise. The parameters for the AD account (attributes like name, location, jobtitle, manager etc) to the runbook can be passed from a SharePoint List in Office 365 which could then be triggered using a Flow.

Summary: I was not able to give you a complete walkthrough but I hope had given some pointers to connect on-premise AD. Forgive my ignorance if I have made some mistakes since I don’t have much experience with IT infrastructure. Hope you find this post useful & informational. Let me know if there is any comments or feedback below.

Execute SharePoint Online PowerShell scripts using Power Automate

Most of us would have used PowerShell for SharePoint to manage SharePoint settings at the organization level and site collection level. SharePoint Online PowerShell commands are very efficient for batch operations for e.g creating multiple sites, list items etc. To use the SharePoint Online PowerShell commands

You must have the SharePoint Admin role or Global Administrator role in Office 365
Install the SharePoint Online Management Shell module

As you know you must be administrator to install a PowerShell module on your workstation which not everyone will have in corporate environments.

I often use a PowerShell script to enable App Catalog at a site collection level to test the PnP webparts & extensions before deploying at the tenant level app catalog based on requirement. If you are not an SPO admin then the dependency is with the SPO admin. In this blogpost I am going to show you how to automate this process by executing PowerShell script to enable App catlog in Azure using Power Automate.

Pre-requisite & permissions:

SPO Admin
Azure Subscription to create Automation account
Access to Premium connector (Azure Automation) in Power Automate
SharePoint List to collect details about the site which needs to have App catalog enabled

To complete this automation process, create the following two components

Automation account in Azure with a Run Book to execute PowerShell script for enabling App Catalog in SP site
Power automate flow to call the Run Book

Automation account in Azure with a Run Book to execute PowerShell script for enabling App Catalog in SP siteAutomation service in Azure is a cloud-based automation and configuration service that supports consistent management across your Azure and non-Azure environments. Go through the documentation from Microsoft to know about this powerfull service in Azure. Let’s use the service in Azure to create a simple Runbook with PowerShell code to enable App catalog in SPO site, you can do much more than this using this service. Refer to this link for the pricing details for the automation service in Azure.

Step 1: Go the Azure portal & create a resource Automation

Enter the name of the automation account, select the Subscription & resource group & click Create

Step 2: After the resource is created, go to the resource & click Modules Gallery under the section Shared Resources as shown below to add the PS SPO module

Search with the keyword “SharePoint” & click “Microsoft.Onlie.SharePoint.PowerShell” and then click Import. This step will the add the SharePoint online PowerShell module for us to use the available PS SPO cmdlets in Runbook.

Now click modules & verify if the SPO PowerShell is added & available.

Step 3: The next step is to add the user credentials (Username & Password) of the SPO admin which is safe & secure by not hardcoding the password on the Runbook. You can also use certificates or AppID AppSecret in PnP online Powershell for creating connection to SPO.

Step 4: Now we are good to create the Runbook, to create it click Runbooks under the section Process Automation and then click Create a runbook. Enter the Name of the Runbook, select the Runbook type to PowerShell and click Create.

Now let’s add the code by editing the runbook to enable app catalog. The section Dynamic Parameters on the code will be passed from flow. To connect to SharePoint Online we are using the SPO admin credentials created in the previous step. Find the code below

# Dynamic Parameters
param(
  [parameter(Mandatory=$true)]
  [string]$SiteURL = "https://domain.sharepoint.com/sites/contosoportal",
  [parameter(Mandatory=$false)]
  [boolean]$enableAppCatalog = 1
)
# Credentials
$myCred = Get-AutomationPSCredential -Name "SPOAdminCred" 
# Parameters
$AdminSiteURL = "https://domain-admin.sharepoint.com"
# Connect to SharePoint Online
Connect-SPOService -Url $AdminSiteURL -Credential $myCred 
# Get the Site Collection
$Site = Get-SPOSite -Identity $SiteURL 
# Enable App catalog
if($enableAppCatalog)
{Add-SPOSiteCollectionAppCatalog -Site $Site}
# Disable App catalog if false
else{Remove-SPOSiteCollectionAppCatalog -Site $Site}
# Get Site Collection Title
Write-Output $Site.Title

Power automate flow to call the Run Book

You can now create a flow with automated trigger from a SharePoint list to get the site url & Boolean value either to enable or disable the app catalog on the site. Here I will be using an Instant flow with trigger “Manually trigger a Flow”

Once the flow is created, add the action “Create Job” under the connector “Azure Automation” which is a premium connector.

Select the Azure Subscription which has the Automation account resource with runbook>Select Resource Group>Select Automation Account>Select the Runbook name which has PS script to enable app catalog. If there is a need to wait until the automation job completes then select Yes on the field “Wait for Job”. For the dynamic parameter, write a JSON to pass the mandatory & optional parameters to the runbook script. On this example I will be passing the Site URL & Boolean value to either enable or disable app catalog using JSON as below

{
  "SiteURL": "https://domain.sharepoint.com/sites/MyFirstTeam",
  "enableAppCatalog": 1
}

If using a SharePoint list, construct the above JSON dynamically with the URL

For the runbook parameters, you might also get an interface as shown below to pass the values (Site Url & enableAppcatalogbooleanvalue).

The flow is ready, run it to test now with parameters.

Summary: The use case I’ve chosen is a simple one but azure automation can be a more powerful service to perform various automation tasks. Find few below

Write Python script in the Runbook
Many samples are available within the Runbook gallery (Create AD user, Display All provisioned site collections etc) under the section Process automation.
With the PowerShell type Runbook all the PS modules (Azure AD for automating AD account creation, PNP SP Online etc) are readily available for us to import easily.
There is a feature by name “Hybrid Runbook Worker Feature” available within Azure Automation account for us to connect Onpremise resources in Azure (e.g SharePoint Onpremise, Onpremise AD etc).
Create a Webhook to call the runbook from an External application by making a POST call
Call a custom built dll by importing them in to the Modules section
Create graphical Runbook with GUI to add cmdlets & to configure the steps
Create schedule linking a runbook

Hope you have enjoyed reading this post and find it useful. If you have any comments or feedback, please provide it on the comments section below.

Access on-premise data in Power Platform & Azure Apps (Logic Apps, Analysis Services & Azure Data factory):

High Availability data gateway setup:

Data gateway architecture:

Integration Service Environment:

Programmatically access your on-premise resources in your Azure Function app

Hybrid Connections:

VNet Integration:

Access on-premise resources in Azure automation account:

Expose your on-premise Application or an existing WEB API in Office 365 cloud:

Share this:

Like this:

To generate a .pnp package (Site Template) from the source site

Azure Automation Runbook:

Power automate flow to call the Run Book:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this: