Working with BizTalk and Octopus Deploy Part 2

Further to my post about deploying BizTalk solution using Octopus found here, this post will describe the substitution of variables in Octopus 3.0 release. In Octopus 2.x.x release, we had Octopus.Platform.dll which gets the variable information from the octopus database, however in octopus 3.0 this feature is removed and not included any more.

This will prevent us to use “Substitute variables in files” process template available on Octopus community. To overcome this issue, I’ve to write custom power shell script to replace variable. Yes, there is in-built substitute variable feature in octopus, apparently we cannot use it, because BizTalk deployment is slightly different from .Net.

Here is the script to substitute variables in files. You need to include all the variables in the below power shell script.

This is the custom variable replacement function”:

$TargetFile=”C:\Program Files (x86)\ABC for BizTalk\1.0\Deployment\EnvironmentSettings\SettingsFileGenerator.xml”
Function ReplaceInFile($TargetFile, [HashTable] $Values){

if ( (Test-Path $TargetFile ) -eq $false){
throw “The target file ‘$($TargetFile)’ does not exist.”
}

Write-Host ” — Starting custom transformation for $($TargetFile)”

$fileContent = Get-Content $TargetFile
$Values.GetEnumerator() | ForEach-Object {
Write-Host “Replacing [$($_.Key)] with [$($_.Value)]”
$fileContent = $fileContent -replace $_.Key, $_.Value
}

[IO.File]::WriteAllText($TargetFile, ($fileContent -join “`r`n”))
}

This is how it is called:

$SitesConfigFile = $TargetFile
ReplaceInFile -TargetFile $SitesConfigFile -Values @{
‘#{SsoAppUserGroup}’ = $SsoAppUserGroup;
‘#{SsoAppAdminGroup}’ = $SsoAppAdminGroup;
}

If someone has a better approach then please let me know.

Thanks,

Shadab Anwer

Advertisements

Using BizTalk 2013 R2 to transform an XML message element to a JSON array

Connected Pawns

I wanted to know whether the out of the box  JSON encoder pipeline component creates an JSON array  if there is only one repeating XML fragment. One of my colleagues found with BizTalk 2013 he had to add an array attribute to the XML to force the creation of a JSON array (https://connectedcircuits.wordpress.com/2014/03/27/sending-json-messages-from-a-biztalk2013-adaptor/).

I created a test schema shown below and deployed it to my run time.

image

I created this message and consumed it with a receive location that was configured with  passthrureceive location.

<ns0:Repeater xmlns:ns0=”http://JSONTest.Repeater”>
<Repeat>Repeat_0</Repeat>
</ns0:Repeater>

The XML message was then processed on a send port with a pipeline that only contained the JSON encoder pipeline component. Voila! I got the correct output shown below;

{
“Repeater”: {
“Repeat”: [
“Repeat_0”
]
}
}

Thanks Microsoft this has saved me a lot of extra work that i had to do in BizTalk 2013.

View original post

Sending JSON messages from a BizTalk2013 adaptor

connectedcircuits

I had a requirement where I needed to send a message to a web API that only accepted messages as JSON.

BizTalk2013 provides a WCF-WebHttp adaptor for sending sending XML messages to a REST endpoint but not in JSON format. There are rumours this will be remedied in BizTalk2013R2, unfortunately I required a solution now.

So in the meantime I will use the the Json.NET component to convert the XML message to JSON. I also found this blog from http://blog.quicklearn.com/2013/09/06/biztalk-server-2013-support-for-restful-services-part-45/ to convert XML messages to JSON using the Json.net in a custom pipeline component which I ended up using with some mods.

When using this pipeline remember to set the Outbound HTTP Headers as shown below:

image

This all worked a treat until during the unit testing phase I created a sample XML document with only one repeating element.This caused the following error from the Web API.

Can not deserialize instance…

View original post 614 more words

Working with BizTalk and Octopus Deploy!!!

In this post, I will demonstrate BizTalk solution deployment using BTDF and OctopusDeploy. Octopus Deploy is a friendly deployment automation system mainly for .NET developers. This post mainly describe BizTalk solution deployment using Octopus deploy.

Some of the pre-requisites are:

  • Octopus Tentacles been installed and configure on the BT server. A “bizTalk-app” (it can be any name) role is created.
  • Octopus Tentacles service running under BT Service account.
  • Environments (BT Server configuration) is created and completed on the Octopus server using Octopus portal. The “green tick” confirm that octopus server can communicate with the BT Machine.

image

  • In BTDF project Environment Settings file template, create a column named “Export_OctopusSettings.xml” before generating MSI as per below.

image

  • Generate MSI using TFS Build. Normally the build server create incremented folders for the generated BTDF MSI. We need to copy the MSI from the drop TFS folder to the local folder on machine. To copying the MSI from the drop TFS folder to the local folder, I created the PS script following the below “Create Step Templates” step below. This script get the latest folder from the drop location and copy the MSI file to the local folder.

$TFSDropDir = $OctopusParameters[“TFSRootDir”]
$TargetFolder = $OctopusParameters[“DestinationFolder”]

if ([string]::IsNullOrEmpty($TFSDropDir)) {
Write-Host “No TFSDropDir has been specified”
Exit
}

if ([string]::IsNullOrEmpty($TargetFolder)) {
Write-Host “No TargetFolder has been specified”
Exit
}
if ( (Test-Path $TargetFolder) -eq $false) {
Write-Host “The target directory doesn’t exist. Unable to copy!”
Exit
}

if ( (Test-Path $TFSDropDir) -eq $false) {
Write-Host “The TFS Drop directory doesn’t exist. Unable to copy!”
Exit
}

$TFSCurrentBuildDir = Get-ChildItem -Path $TFSDropDir -name | Sort-Object LastAccessTime -Descending | Select-Object -First 1
$DeployMSICompletePath= “$TFSDropDir\” + “$TFSCurrentBuildDir\”
Write-Host  “MSI Full path: $DeployMSICompletePath”
# Copy the items!
Write-Host “Copying items from $DeployMSICompletePath to $TargetFolder … please wait.”
robocopy $DeployMSICompletePath $TargetFolder

#Robocopy can success with multiple codes!
$exitcode = $LastExitCode

if($exitcode -in 0,1,2,3,4,5,6,7){
$LastExitCode = 0
}

return $LastExitCode

  • “Run – Windows Installer” and “Variable – Substitute in files step” templates should be created in octopus (to create/import templates see below). You can download these templates from octopus library https://library.octopusdeploy.com/#!/listing

Steps to follow on Octopus Server.

  1. Create Step Templates for BTDF Deployment using the below script. You just need to Import the script using the Import button on the Step templates page. Copy the below script in the below window.

image

For Template please contact Shadab.Anwer@SharpTalkTech.com

The above Json template will generate the below PS script.

For PS script please contact Shadab.Anwer@SharpTalkTech.com

2. Save the templates and go to the Parameters tab on the same screen. All parameters can be viewed as per the below screen.

image

3. Navigate to created Project “Test”. Click “Process”, click “add Step”. Chose “Run – Windows Installer” from the step type.

image

4. Select target roles “(this is the role created during configuration of octopus tentacles”).  MSI File Path “C:\Octopus\Test.MSI”. Click Save. If you using nuget package (.nuspec) step to extract MSI on Octopus tentacles folder, you can use the below to run the MSI.

#{Octopus.Action[Deploy Package].Output.Package.InstallationDirectoryPath}\#{Octopus.Project.Name}-#{Octopus.Release.Number}.msi

5. Click “Add Step” to replace variables in the setting file using Octopus variables (mentioned below). Chose “Variables – Substitute in Files” from the step type. Parameter template files should be “C:\Program Files (x86)\Test for BizTalk\1.0\Deployment\EnvironmentSettings\SettingsFileGenerator.xml”. Click Save.

image

6. Click “Add Step” to create BTDF deployment. Chose “Deploy – BizTalk Application” from the step type. Below is the configuration details.

image

Install Directory – #{Octopus.Action[Run – Windows Installer].Output.InstallDirectory}

Package Install – #{Octopus.Action[Deploy Package].Output.Package.InstallationDirectoryPath}

Process page should look like the below.

image

7. Navigate to Variables. Create Variables as mentioned in the below screen. These variables can be distinguished on basis of scope. You can create many variables with the same name for different scope (or environment like TEST, QA, Prod etc).

image

8. Next step is to create Release.  Goto Release, click create release and save. Deploy this release to the specific server and can later promote this release to other servers.

image

9. Click on Task Log – there is all the information including the logging of the deployment steps. The BTDF deployment logging is also in there.

image

BizTalk Continuation Integration Build and Deployment can be fully automated using Teamcity and Octopus, you need to install octopack to do so. At the moment, I am working with automating CI with TFS and Octopus. If someone has already done this, would love to hear from them.

Thanks,

Shadab Anwer

Cannot perform encryption or decryption because the secret is not available from the master secret server

When I tried to open any configuration for send port/Received location I use bump into this error: “Cannot perform encryption or decryption because the secret is not available from the master secret server”. A simple way to solve this problem:

  • Open a command prompt
  • Navigate to C:\Program Files\Common Files\Enterprise Single Sign-On\
  • Type in this command: ssoconfig -restoresecret SSO3F38.bak (or another file that looks like this one, it’s the SSO back-up file)
  • Enter the password (This is the password for SSO created during BizTalk Configuration)
  • Create a new back-up file with this command: ssoconfig -backupsecret latestbackup.dat

Thoughts on Integration Architecture and Analysis.

Apologies for not being a regular blogger these days, this is my first post from Australia. I have joined new company and moved to Brisbane – a beautiful Queensland city.

This post is mainly related to the Integration competency where I am just going to share good work being done by the architect(s) to make feel better and speed up the new comer(s) in the company like myself. I have been lacking with the terms in the previous engagement(s) like Business Service, Business Events, Integration points, Product Adapters, Channel Adapters Etc…Etc… These are key components of an Integration projects and need to be well define in the analysis phase. A proper maintenance in Wiki shall be ideal for people to understand the Integration solution.

The steps which shall be followed in the Architecture/Analysis phase are:

  • Define Business Services/Business Events. For eg: If the message needs to be send from Party A to Party B to create purchase order. A Business service needs to be created like “PurchaseOrderManagement” and Business Events need to be define as CreatePurchaseOrder(), UpdatePurchaseOrder(), RetreivePurchaseOrder etc.

Creating Wiki page or register where you can add the top level “Business Services” and sub levels Business Events               (or operations) shall be ideal. Give definition to your Business services and include all the business events.

Screen for the business service register should look like below:

Screen

The name of the service register should be “PurchaseOrderManagement”. The service operations are the different              events which will act as a business service. Like CreatePurchaseorder, UpdatePurchaseOrder,                                RetreivePurchaseOrder etc.

  • Each business service should have its own page to describe and define message exchange patterns 1-way, 2-way request-response etc, Integration Points, Sequence diagram, data contract, Faults (business/system), request message schema and sample, response message schema and sample, end points etc.

The page should look like this:

Screen1

  • Integration point diagram (component diagram) and sequence diagram should be linked or added to the service operation page.
  • Integration Point Diagram: – This is a component diagram reflecting the integration points. These integration points are in the integration layer and needs to be developed in ESB (WCF/BizTalk).

Screen2

Screen3

The above diagram shows the front end caller, Integration Layer with the channel adapter and the ESB business                   service and end systems. On the same page there is a technical design section to provide the links to the business               service operations this integration point is link to.

The architecture and analysis phase are more important in the life cycle of the integration project, there are many chances that things can go wrong if proper thoughts and consideration is not taken into account, otherwise we might end up doing re-factoring or re-designing the solution.

Thanks.

Skip Create BizTalk Application using BTDF

This article is to demonstrate how to skip create BizTalk Application using BTDF. By default the btdf project will call the target “DeployAppDefinition”. This target is responsible to create BizTalk Application when it is not exists in the Admin Console, otherwise it throw an error when the application is found.

In my current project we were require to only deploy the version assemblies for the particular application without removing the BT application.

I came up with the below BTDF Project which is using the custom target to call DeployAppDefinition which checks for the existence of the application and do not remove if exists.

I need to create this variable in the property group “<SkipRemoveApp>True</ SkipRemoveApp >  “ and used it in the DeployAppDefinition target.

<!– Skip BizTalk application definition –>

<Target Name=”DeployAppDefinition” Condition=”‘$( SkipRemoveApp)’ == ‘False'”>

<Exec Command=”BTSTask.exe AddApp -ApplicationName:&quot;$(BizTalkAppName)&quot; -Description:&quot;$(ProjectName)&quot;” />

<AddAppReference ApplicationName=”$(BizTalkAppName)” AppsToReference=”@(AppsToReference)” Condition=”%(Identity) == %(Identity) and ‘@(AppsToReference)’ != ”” />

</Target>

Below is the complete btdf project. Please let me know if anyone having a different approach.

<?xml version=”1.0″ encoding=”utf-8″?>

<!–

***************************************************

**** DEPLOYMENT STEPS *****************************************************************************************************

This deployment package will only add BizTalk resources to the existing application.

***************************************************************************************************************************

–>

<Project xmlns=”http://schemas.microsoft.com/developer/msbuild/2003&#8243;

DefaultTargets=”Installer”

ToolsVersion=”4.0″

TreatAsLocalProperty=”SkipUndeploy”> <!– **** Add this so this variable is accessible on this target file ****–>

<PropertyGroup>

<Configuration Condition=”‘$(Configuration)’ == ””>Debug</Configuration>

<Platform Condition=”‘$(Platform)’ == ””>x86</Platform>

<SchemaVersion>1.0</SchemaVersion>

<ProjectName>ESB.ReferenceSystems.P</ProjectName>

<ProjectVersion Condition=”‘$(ProjectVersion)’ == ””>1.1</ProjectVersion>

<ProjectVersion Condition=”‘$(ProjectVersion)’ != ””>$(ProjectVersion)</ProjectVersion>

<IncludeSchemas>False</IncludeSchemas>

<IncludeOrchestrations>False</IncludeOrchestrations>

<IncludeTransforms>False</IncludeTransforms>

<IncludePipelines>False</IncludePipelines>

<IncludeComponents>False</IncludeComponents>

<IncludePipelineComponents>False</IncludePipelineComponents>

<IncludeCustomFunctoids>False</IncludeCustomFunctoids>

<IncludeVocabAndRules>False</IncludeVocabAndRules>

<IncludeVirtualDirectories>False</IncludeVirtualDirectories>

<IncludeMessagingBindings>False</IncludeMessagingBindings>

<IncludeDeploymentTest>False</IncludeDeploymentTest>

<Includelog4net>False</Includelog4net>

<IncludeSSO>False</IncludeSSO>

<IncludeEsbItineraries>False</IncludeEsbItineraries>

<IncludeBam>False</IncludeBam>

<IncludeInstallUtilForComponents>False</IncludeInstallUtilForComponents>

<UsingMasterBindings>True</UsingMasterBindings>

<RequireXmlPreprocessDirectives>False</RequireXmlPreprocessDirectives>

<ApplyXmlEscape>True</ApplyXmlEscape>

<IncludeSettingsSpreadsheetInMsi>True</IncludeSettingsSpreadsheetInMsi>

<SkipIISReset>True</SkipIISReset>

<SkipHostInstancesRestart>False</SkipHostInstancesRestart>

<StartApplicationOnDeploy>True</StartApplicationOnDeploy>

<EnableAllReceiveLocationsOnDeploy>False</EnableAllReceiveLocationsOnDeploy>

<StartReferencedApplicationsOnDeploy>False</StartReferencedApplicationsOnDeploy>

<UseIsolatedAppDomain>False</UseIsolatedAppDomain>

<EnableBizTalkExtendedLogging>False</EnableBizTalkExtendedLogging>

<EnableBizTalkAssemblyValidation>False</EnableBizTalkAssemblyValidation>

<EnableBizTalkCorrelationValidation>False</EnableBizTalkCorrelationValidation>

<EnableBizTalkSchemaValidation>False</EnableBizTalkSchemaValidation>

<SkipAddApp>True</SkipAddApp>

</PropertyGroup>

<PropertyGroup>

<!– Properties related to building an MSI for server deployments –>

<!– BizTalk App Version Upgrade –>

<!–   For each new product release to be deployed to your BizTalk servers: –>

<!–     1) Increment ProductVersion –>

<!–     2) Generate a new GUID and update ProductId with the new GUID –>

<!–   This allows the new MSI to automatically uninstall (not undeploy!) the old MSI and install the new one. –>

<ProductVersion Condition=”‘$(ProductVersion)’ == ””>1.0.1</ProductVersion>

<ProductVersion Condition=”‘$(ProductVersion)’ != ””>$(ProductVersion)</ProductVersion>

<ProductId Condition=”‘$(ProductId)’ == ””>A66F460E-7B65-4330-843B-477251BF3AEC</ProductId>

<ProductId Condition=”‘$(ProductId)’ != ””>$(ProductId)</ProductId>

<!– BizTalk App Version Upgrade –>

<ProductName>ESB.ReferenceSystems.P for BizTalk</ProductName>

<Manufacturer></Manufacturer>

<PackageDescription>ESB.ReferenceSystems.P</PackageDescription>

<PackageComments>ESB.ReferenceSystems.P</PackageComments>

<!– NEVER change the ProductUpgradeCode. –>

<ProductUpgradeCode>A87DEA9B-53B5-4A30-85E3-71F5C698D203</ProductUpgradeCode>

<IsInTFSBuildMode Condition=”Exists(‘$(OutDir)\$(Configuration)\ESB.ReferenceSystems.P.SubmissionProcessAPI.Schemas.dll’)”>True</IsInTFSBuildMode>

</PropertyGroup>

<!– Under TFS Team Build, set CustomizableOutDir property to true in TFS 2005/2008/2010 UpgradeTemplate. –>

<!– With a workflow build, copy the default template then modify the MSBuild task for the solution build. Set OutDir to blank and –>

<!– CommandLineArguments to String.Format(“/p:SkipInvalidConfigurations=true;TeamBuildOutDir=””{0}”” {1}”, BinariesDirectory, MSBuildArguments). –>

<PropertyGroup Condition=”‘$(Configuration)’ == ‘Debug'”>

<DeploymentFrameworkTargetsPath>$(MSBuildExtensionsPath)\DeploymentFrameworkForBizTalk\5.0\</DeploymentFrameworkTargetsPath>

<OutputPath Condition=”‘$(TeamBuildOutDir)’ == ””>bin\Debug\</OutputPath>

<OutputPath Condition=”‘$(TeamBuildOutDir)’ != ””>$(TeamBuildOutDir)</OutputPath>

<DeployPDBsToGac>false</DeployPDBsToGac>

</PropertyGroup>

<PropertyGroup Condition=”‘$(Configuration)’ == ‘Release'”>

<DeploymentFrameworkTargetsPath>$(MSBuildExtensionsPath)\DeploymentFrameworkForBizTalk\5.0\</DeploymentFrameworkTargetsPath>

<OutputPath Condition=”‘$(TeamBuildOutDir)’ == ””>bin\Release\</OutputPath>

<OutputPath Condition=”‘$(TeamBuildOutDir)’ != ””>$(TeamBuildOutDir)</OutputPath>

<DeployPDBsToGac>false</DeployPDBsToGac>

</PropertyGroup>

<PropertyGroup Condition=”‘$(Configuration)’ == ‘Server'”>

<DeploymentFrameworkTargetsPath>Framework\</DeploymentFrameworkTargetsPath>

<!– Get our PDBs into the GAC so we get file/line number information in stack traces. –>

<DeployPDBsToGac>true</DeployPDBsToGac>

</PropertyGroup>

<ItemGroup>

<PropsFromEnvSettings Include=”SsoAppUserGroup;SsoAppAdminGroup” />

</ItemGroup>

<ItemGroup>

<AdditionalFiles Include=”Deployment.config”>

<LocationPath>.</LocationPath>

</AdditionalFiles>

</ItemGroup>

<ItemGroup>

<!–<AppsToRemove Include=”ESB.ReferenceSystems.P” />–>

</ItemGroup>

<!– !!! TODO !!! –>

<Import Project=”$(DeploymentFrameworkTargetsPath)BizTalkDeploymentFramework.targets” />

<!–

The Deployment Framework automatically packages most files into the server install MSI.

However, if there are special folders that you need to include in the MSI, you can

copy them to the folder $(RedistDir) in the CustomRedist target.

To include individual files, add an ItemGroup with AdditionalFiles elements.

–>

<!–

The Targets below are to enable undeploy during the TFS build when not deployed – Bill Chesnut

–>

<!– **** PRE DEPLOYMENT SECTION ******************************************************************************************–>

<!– **********************************************************************************************************************–>

<!– **** The BTDF out of the box calls this target before it deploys the application                                  ****–>

<!– **** Set the “SkipUndeploy” to false so that we can undeploy the application before deployment                    ****–>

<!– **** Then call the Custom Deploy Target to add the undeployment step                                              ****–>

<!– **********************************************************************************************************************–>

<Target Name=”CustomDeployTarget”>

<PropertyGroup>

<SkipUndeploy>false</SkipUndeploy>

</PropertyGroup>

<CallTarget Targets=”CustomDeploy” />

</Target>

<!– **********************************************************************************************************************–>

<!– **** This target is called from CustomDeployTarget                                                                ****–>

<!– **** DeployAppDefinition is an out of the box target  which will skip the check for Add Application               ****–>

<!– **** CustomUndeployApp is the custom target created to undeploy the application                                   ****–>

<!– **** CustomDeployAssemblyTarget is the custom target to Add the BizTalk Assemblies                                ****–>

<!– **********************************************************************************************************************–>

<Target Name=”CustomDeploy” DependsOnTargets=”$(CustomDeployTargetDependsOn)” />

<PropertyGroup>

<!– CustomDeploy depends on this CustomDeployTargetDependsOn –>

<CustomDeployTargetDependsOn>

DeployAppDefinition;

CustomDeployAssemblyTarget;

</CustomDeployTargetDependsOn>

</PropertyGroup>

<!– Skip BizTalk application definition –>

<Target Name=”DeployAppDefinition” Condition=”‘$(SkipAddApp)’ == ‘False'”>

<Exec Command=”BTSTask.exe AddApp -ApplicationName:&quot;$(BizTalkAppName)&quot; -Description:&quot;$(ProjectName)&quot;” />

<AddAppReference ApplicationName=”$(BizTalkAppName)” AppsToReference=”@(AppsToReference)” Condition=”%(Identity) == %(Identity) and ‘@(AppsToReference)’ != ”” />

</Target>

<Target Name=”CustomDeployAssemblyTarget” DependsOnTargets=”$(CustomDeployAssemblyTargetDependsOn)” />

<PropertyGroup>

<!– CustomPostDeployTarget depends on this CustomPostDeployTargetDependsOn –>

<CustomDeployAssemblyTargetDependsOn>

AddBizTalkResources;

</CustomDeployAssemblyTargetDependsOn>

</PropertyGroup>

<!– **********************************************************************************************************************–>

<!– **** This target is responsible for adding all the biztalk resources into the application resources and is called ****–>

<!– **** from CustomDeployAssemblyTarget                                                                                     ****–>

<!– **********************************************************************************************************************–>

<Target Name=”AddBizTalkResources” DependsOnTargets=”$(AddBizTalkResourcesDependsOn)” />

<PropertyGroup>

<!– AddBizTalkResources depends on this AddBizTalkResourcesDependsOn –>

<AddBizTalkResourcesDependsOn>

AddBizTalkAssembliesToBizTalkResources;

</AddBizTalkResourcesDependsOn>

</PropertyGroup>

<!– **********************************************************************************************************************–>

<!– **** This target is responsible for adding the BizTalk assemblies into the application resources and is called       ****–>

<!– **** from AddBizTalkResources                                                                                     ****–>

<!– **********************************************************************************************************************–>

<Target Name=”AddBizTalkAssembliesToBizTalkResources”>

<Message Text=”In AddBizTalkAssembliesToBizTalkResources.-2″/>

<Exec

Command=”BTSTask.exe AddResource -Type:BizTalkAssembly  -Source:&quot;..\SubmissionProcessAPI.Schemas\bin\Debug\ESB.ReferenceSystems.P.SubmissionProcessAPI.Schemas.dll&quot; -ApplicationName:&quot;$(BizTalkAppName)&quot; -Overwrite -Options:GacOnAdd,GacOnImport,GacOnInstall”

Condition=”‘$(DeployBizTalkMgmtDB)’ == ‘true’ ” />

</Target>

</Project>