Integration Scatter-Gather pattern using Azure Logic Apps and Service Bus

connectedcircuits

Recently I have taken a role as the Integration Architect and tech lead for a project to integrate messaging between MS Dynamics 365 AX/CRM and 3rd party systems. This was a huge solution that took over 10 months to design and develop. It consisted of developing over 60 Logic Apps.

With a large integration project like this were multiple entities are required to be updated across multiple systems in a transactional manner, I required a process to correlate and aggregate all the responses into one composite message before proceeding onto the next task in the workflow. The scatter-gatherer pattern was chosen as a good candidate for this type of scenario.

The next problem was how to implement this pattern using Logic Apps. Using what I had learnt from my previous blog on enforcing ordered delivery of messages https://connectedcircuits.wordpress.com/2017/08/26/enforcing-ordered-delivery-using-azure-logic-apps-and-service-bus/ I was able to design the solution below.

image

It is based on…

View original post 1,556 more words

Advertisements

Message Re-sequencing using Azure Functions and Table storage

connectedcircuits

Message re-sequencing is another integration pattern used to rearrange messages received out-of-order and back into the original sequence again. Well you may ask yourself why is this pattern is so important in todays world of messaging systems.

As the integration world is moving towards services hosted as Microservices in the cloud due to its scalability attributes and independent deployable components. This may require breaking messages up into smaller logical segments or de-batching the messages to allow the services to manage the data more efficiently. Using this pattern, makes it possible to add the smaller segments back together again after being broken up in its correct order.

image

Scenario

A typical use case is a Purchase Order with multiple line items which require processing asynchronously by a Microservice at an individual line level. This service may calculate the taxes, check stock levels and add the total line price. Some line items may…

View original post 2,691 more words

Enforcing Ordered Delivery using Azure Logic Apps and Service Bus

connectedcircuits

When consuming messages from an Azure service bus the order may not be guaranteed due to the brokered based messaging scheme where multiple consumers can consume messages from the bus. Sure you can force the Logic App to execute as a single instance but then you sacrifice performance and scalability. You can also use ReceiveAndDelete but then you loose the transactional nature of the bus. Ultimately to ensure a message is consumed in the correct order using the transactional nature of the bus you would add a sequence number to each message and use this to enforce the ordering.

To achieve ordered delivery using Logic Apps, you would need to ensure all related messages are consumed by the same Logic App instance and for this we use the session Id property on the service bus. Below is the full workflow process to force ordered delivery using Logic Apps and session…

View original post 592 more words

Itinerary based message routing using Azure Logic Apps and Service Bus Actions

connectedcircuits

This is another integration pattern used quite extensively in the integration world. It is used when a message is required to be routed to several endpoints in a particular order using some form of routing list. Depending on your business requirements, the message being routed may be enriched or replaced before sending it to the next service endpoint in the list.

image

This is probably one of the easiest integration patterns to implement using Logic Apps and Service Bus Actions. There are probably other methods to implement this pattern, but I wanted to abstract the routing logic away from the Logic App itself and leave it to focus on the business process and not worry about setting up the next routing endpoint.

By using the Actions feature of the service bus I can defined the routing order of the next service endpoint, or in this case the next service bus subscription…

View original post 840 more words

Content based message routing using Azure Logic Apps, Function and Service Bus

connectedcircuits

Content Based Routing (CBR) is another pattern used in the integration world. The contents of the message determines the endpoint of the message.

This article will describe one option to develop this pattern using an Azure Service Bus, an Azure Function and a Logic App.

Basically the service bus will be used to route the message to the correct endpoint using topics and subscriptions. The Azure Function is used to host and execute the business rules to inspect the message contents and set the routing properties. A logic app is used to accept the message and call the function passing the received message as an argument. Once the function executes the business rules, it will return the routing properties in the response body. The routing information is then used to set the properties on the service bus API connector in the Logic App before publishing the message onto the bus.

View original post 1,060 more words

Validating Json Schema in Azure Logic Apps

Hi All,

In this post we will show how we can validate Json schema against the message  in Logic Apps. I and Krishna Pochanapeddi working on building an interface service in Logic Apps. We need to send a request message to Logic Apps request connector. We need to validate the same message against the Json schema. There is no capability within Logic Apps to validate the names of the fields in the Json message. We can do this easily using Azure Function by passing the Schema and request message using Azure Function connector. However, we do not want to use Azure function to validate schema. We want to use Logic Apps as a complete solution for these validation issues. It is easy to validate XML Schema using Integration Account but Json message cannot be validated.

After wondering for few hours; reading Json best practices; we  found the basic and powerful Json capability which is the object option “required”. Now we can mentioned the required fields (field names) in the schema itself.

We created below mentioned Json schema

{
“$schema”: “http://json-schema.org/draft-04/schema#”,
“definitions”: {},
“id”: “http://example.com/example.json”,
“properties”: {
“ChangePasswordRequest”: {
“id”: “/properties/ChangePasswordRequest”,
“properties”: {
“CurrentPassword”: {
“default”: “currenthashedpassword”,
“description”: “An explanation about the purpose of this instance.”,
“id”: “/properties/ChangePasswordRequest/properties/CurrentPassword”,
“title”: “The currentpassword schema”,
“type”: “string”
        },
“Identifier”: {
“default”: “126”,
“description”: “An explanation about the purpose of this instance.”,
“id”: “/properties/ChangePasswordRequest/properties/Identifier”,
“title”: “The identifier schema”,
“type”: “string”
        },
“IdentifierScheme”: {
“default”: “test”,
“description”: “An explanation about the purpose of this instance.”,
“id”: “/properties/ChangePasswordRequest/properties/IdentifierScheme”,
“title”: “The identifierscheme schema”,
“type”: “string”
        },
“MessageIdentifier”: {
“default”: “f7b351fb-ade4-4361-bfc3-9bb7df783880”,
“description”: “An explanation about the purpose of this instance.”,
“id”: “/properties/ChangePasswordRequest/properties/MessageIdentifiers”,
“title”: “The messageidentifiers schema”,
“type”: “string”
        },
“MessageTimeStamp”: {
“default”: “2016-04-04T14:15:02.6476354+10:00”,
“description”: “An explanation about the purpose of this instance.”,
“id”: “/properties/ChangePasswordRequest/properties/MessageTimeStamp”,
“title”: “The messagetimestamp schema”,
“type”: “string”
        },
“NewPassword”: {
“default”: “Pass126”,
“description”: “An explanation about the purpose of this instance.”,
“id”: “/properties/ChangePasswordRequest/properties/NewPassword”,
“title”: “The newpassword schema”,
“type”: “string”
        },
“NotificationAddress”: {
“default”: “kcpochanapeddi@gmail.com”,
“description”: “An explanation about the purpose of this instance.”,
“id”: “/properties/ChangePasswordRequest/properties/NotificationAddress”,
“title”: “The notificationaddress schema”,
“type”: “string”
        },
“NotificationPreference”: {
“default”: “Email”,
“description”: “An explanation about the purpose of this instance.”,
“id”: “/properties/ChangePasswordRequest/properties/NotificationPreference”,
“title”: “The notificationpreference schema”,
“type”: “string”
        },
“OriginatingSystem”: {
“default”: “test”,
“description”: “An explanation about the purpose of this instance.”,
“id”: “/properties/ChangePasswordRequest/properties/OriginatingSystem”,
“title”: “The originatingsystem schema”,
“type”: “string”
        }
      },
required”: [
“MessageIdentifier”,
“NotificationAddress”,
“CurrentPassword”,
“Identifier”,
“OriginatingSystem”,
“NotificationPreference”,
“NewPassword”,
“IdentifierScheme”,
“MessageTimeStamp”
      ],
“type”: “object”
    }
  },
“required”: [
“ChangePasswordRequest”
  ],
“type”: “object”
}

Added this schema in the Logic app request connector and Parse Json connector.

image

If we send a invalid message  as per below:

{
“ChangePasswordRequest”: {
“MessageIdentifier”: “f7b351fb-ade4-4361-bfc3-9bb7df783880”,
“OriginatingSystem”: “test”,
“MessageTimeStamp”: “2016-04-04T14:15:02.6476354+10:00”,
“Identifie”: “125”, <Invalid field name; valid is Identifier>
“IdentifierScheme”: “test”,
“CurrentPassword”: “currenthashedpassword”,
“NewPassword”: “YGU0PCPEM”,
“NotificationPreference”: “Email”,
“NotificationAddress”: “kcpochanapeddi@gmail.com”
  }
}

The above message fails in the parse Json with the message “Requred properties are missing from object…” as per the below screen:

image

Now this can be handled in Logic Apps using “Add a Parallel branch->Add an Action” as per below screen.

image

Click on eclipses on action “Execute stored procedure” and “Response 4” and click on configure run after to configure the appropriate action.

image

Yay!!! Well Done Krishna Smile.

Regards

Shadab

Creating Azure Virtual Machine from Disk Image and continuous deployment using Bamboo.

In this post I will showing how to use Bamboo to create Azure Virtual machine from existing disk image and to do deployment of the projects/artefacts on the newly created virtual machine. The Idea was to create a Dev/Test environment in the cloud with all the application installed.

  1. Create New Plan in Bamboo.

Screen Shot 2017-07-24 at 3.29.30 pm

2. Click on Configure Plan.

Add Task – Choose Script. Enter the PowerShell script as per the below. The script will ask for Azure Login details, enter your azure subscription login details. The below script will create a txt file VirtualMachine.txt with the value “BizTalkIpAddress=<IPAddress>” of the newly created Virtual machine

Screen Shot 2017-07-24 at 3.33.07 pm

Login-AzureRmAccount

Get-AzureRmSubscription

Select-AzureRmSubscription -SubscriptionId “<your subscription id>”

Set-AzureSubscription -SubscriptionId “<your subscription id”

$resourceGroupName = “Dev-BizTalk-VM-ResourceGroup”

$sourceUri = “https://<storage Name>.blob.core.windows.net/vhds/bt2013R2Lab01btOSDisk.vhd” #Link to your existing disk image vhd file.

$location = “australiasoutheast”

$snapshotName = “bt2013R2Lab01btOSDisk_snapshot”

$StorageName = “btdevstorage”

$StorageType = “StandardLRS”

## Network

$InterfaceName = “btNetworkInterface0” + ${bamboo.buildNumber}

Write-Host “Inteface:”,${bamboo.buildNumber}

$Subnet1Name = “btSubnet01” 

$VNetName = “btVNet01”

$VNetAddressPrefix = “10.0.0.0/16

$VNetSubnetAddressPrefix = “10.0.0.0/24

## Compute

$VMName = “bt2013R2Lab0” + ${bamboo.buildNumber}

$ComputerName = “bt2013R2Lab0” + ${bamboo.buildNumber}

$VMSize = “Standard_DS2_v2”

$OSDiskName = $VMName + “btOSDisk”

$disk = Get-AzureRmDisk -ResourceGroupName $resourceGroupName -DiskName $dataDiskName 

$osDiskName = “bt2013R2Lab0” + ${bamboo.buildNumber} + “btOSDisk”

Write-Host “OSDiskName:”,$osDiskName

$osDisk = New-AzureRmDisk -DiskName $osDiskName -Disk `

    (New-AzureRmDiskConfig -AccountType StandardLRS  -Location $location -CreateOption Import `

    -SourceUri $sourceUri) `

    -ResourceGroupName $resourceGroupName

$storageacc = Get-AzureRmStorageAccount -ResourceGroupName $ResourceGroupName 

# Network

$vnet   = Get-AzureRMVirtualNetwork -Name $VNetName -ResourceGroupName $ResourceGroupName

$pip = New-AzureRmPublicIpAddress -Name $InterfaceName -ResourceGroupName $ResourceGroupName -Location $Location `

   -AllocationMethod Dynamic

$nic = New-AzureRmNetworkInterface -Name $InterfaceName -ResourceGroupName $ResourceGroupName `

    -Location $location -SubnetId $vnet.Subnets[0].Id -PublicIpAddressId $pip.Id -NetworkSecurityGroupId $nsg.Id

$user = “admin”

$password = ”

$securePassword = ConvertTo-SecureString $password -AsPlainText -Force

$Credential = New-Object System.Management.Automation.PSCredential ($user, $securePassword) 

$vmConfig = New-AzureRmVMConfig -VMName $vmName -VMSize “Standard_A2”

$vm = Add-AzureRmVMNetworkInterface -VM $vmConfig -Id $nic.Id

# Create the VM in Azure

$vm = Set-AzureRmVMOSDisk -VM $vm -ManagedDiskId $osDisk.Id -StorageAccountType $StorageType `

    -DiskSizeInGB 128 -CreateOption Attach -Windows

New-AzureRmVM -ResourceGroupName $ResourceGroupName -Location $location -VM $vm

Set-AzureRmVMAccessExtension -ResourceGroupName $resourceGroupName -VMName $VMName `

    -Name $ComputerName -Location $location -UserName $Credential.GetNetworkCredential().Username `

    -Password $Credential.GetNetworkCredential().Password -typeHandlerVersion “2.0”

$net=Get-AzureRmPublicIpAddress -ResourceGroupName $resourceGroupName -Name $InterfaceName

$ipAddress = $net.IpAddress

$Content = “BizTalkIpAddress=$ipAddress”

Write-Host $Content

write-output $Content | add-content ${bamboo.build.working.directory}\Packages\VirtualMachine.txt

3.  Create Deployment Steps.

  • Artifact Download

Screen Shot 2017-07-24 at 3.36.44 pm

  • Inject Bamboo Variables. This is used to get the IP Address of the newly created virtual machine.

Screen Shot 2017-07-24 at 3.36.56 pm

  • Copy Package to the Azure VM. This is used to copy the downloaded packages to the newly created virtual machine.

$biztalkmachineIP =  ${bamboo.Azure.BizTalkIpAddress}

Write-Host “BizTalkIpAddress:”,${bamboo.Azure.BizTalkIpAddress}

$user = “admin”

$password = ”

$securePassword = ConvertTo-SecureString $password -AsPlainText -Force

$Credential = New-Object System.Management.Automation.PSCredential ($user, $securePassword) 

$InstallerFile = “C:\SRC\STT.Common.IaaS.Test\STT.Common.IaaS.Test\MSI\STT.Common.IaaS.msi”

New-PSDrive -Name Y -PSProvider filesystem -Root “\\${bamboo.Azure.BizTalkIpAddress}\C$” -Credential $Credential

Copy-Item $InstallerFile -destination Y:\ -recurse 

Remove-PSDrive Y

#$session = New-PSSession -ComputerName ${bamboo.Azure.BizTalkIpAddress}  -Credential $Credential  -ConfigurationName Microsoft.Powershell32

#$LASTEXITCODE = Invoke-Command -Session $session -ScriptBlock {msiexec.exe /i “C:\STT.Common.IaaS.msi” /passive /log “c:\log.txt”}

Screen Shot 2017-07-24 at 3.37.52 pm

  • Install Application : This script will execute the installer package. This is just to prove that the packages are getting installed on the new machine.

Write-Host “BizTalkIpAddress:”,${bamboo.Azure.BizTalkIpAddress}

$user = “admin”

$password = ”

$securePassword = ConvertTo-SecureString $password -AsPlainText -Force

$Credential = New-Object System.Management.Automation.PSCredential ($user, $securePassword) 

$session = New-PSSession -ComputerName ${bamboo.Azure.BizTalkIpAddress}  -Credential $Credential  -ConfigurationName Microsoft.Powershell32

$LASTEXITCODE = Invoke-Command -Session $session -ScriptBlock {msiexec.exe /i “C:\STT.Common.IaaS.msi” /passive /log “c:\log.txt”}

$exit_status = $LASTEXITCODE

exit $exit_status

Screen Shot 2017-07-24 at 3.37.32 pm

The whole idea is to how create instant environment in the Cloud. Once the environment is created we can download/build the packages from any repository and deploy to the new machine. I’ve just used bamboo because to give visual touch and use as an continuous Integration and deployment.

Note: This is not the BizTalk environment, with BizTalk there is still few things to be done on the machine.

Thanks.