ESB.Extensions

Updated Wiki: Home

BVeldhoen — Mon, 23 Nov 2009 16:38:15 GMT

Project Description
Extends the BizTalk ESB Toolkit 2.0 with a number of generic services such as ReceivePipelineService, SendPipelineService, and ResequencerService. Uses a somewhat different resolution framework to enable better resolution: resolving serializable objects rather than just strings.

More detailed information can be found on my blog about the ESB.Extensions:
http://geekswithblogs.net/BVeldhoen/archive/2009/09/11/debatching-aggregation-and-resequencing-using-the-esb.extensions.aspx
http://geekswithblogs.net/BVeldhoen/archive/2009/08/28/my-beef-with-the-esb-toolkit-2.0-resolution-framework.aspx

The ESB.Extensions solution contains a number of services, that were implemented on top of the ESB Toolkit 2.0. The ESB.Extensions uses a slightly different resolution framework, which allows for resolution of fully-fledged serializable objects rather than just strings. (To allow for resolution of only strings was just too limited and not enough generic to my taste). The ESB.Extensions resolution framework is still compatible with the ESB configuration and itinerary designer.

The following Services (orchestrations) have been included up to now:

ReceivePipelineService: Can be used to execute a receive pipeline from within the orchestration. One obvious application for this service is (post-receive) debatching using the XmlReceive pipeline (or any custom pipeline that includes the XmlDisassembler or FFDisassembler). This service can be configured using the ESB.Extensions resolution framework with the following parameters:
- Receive pipeline type
SendPipelineService: Can be used to execute a send pipeline from within the orchestration. One application for this service is aggregation using pipelines, that contain the XmlAssembler or FFAssembler components. This service, too, can be configured using the ESB.Extensions resolution framework with the following parameters:
- Send pipeline type
- Batch Timeout: The timeout to apply to the entire batch. When this timeout expires, all the collected messages up to that point in time will be aggregated and the service completes.
- Message Timeout: The timeout to apply to each message. When this timeout expires, all the collected messages up to that point in time will be aggregated and the service completes.
- XLANGMessageComparer type: The SendPipelineService uses a serializable list to collect the messages to be aggregated. You can specify the comparer type to control how the messages are sorted. This happens in memory, but does not add much overhead compared to the use of Microsoft.XLANGs.Pipeline.SendPipelineInputMessages.
Resequencer and ResequencerGo services: These two services can be used in conjunction to resequence a given sequence (or batch) of messages.

EsbBatchDb
The EsbBatchDb can be used to track batches, so that batches can be dynamically combined or otherwise defined based upon the resolver implementations. The ESB.Extensions has an orchestration HandleBatchCompleteNotification, which works together with the WCF-SQL adapter to handle notifications, sent by Sql Server. Those notifications are used to generate Go messages, which in turn will be used by the Resequencer(Go) orchestrations to start resequencing the batches.

TODO
A number of issues are still open, if anyone's interested to take on any number of them:

Fix the Itineraries in the designer. They are outdated compared to the exported itineraries in xml, which were manually edited.
Add a pipeline component that registers an incoming batch (in a database? EsbBatchDb?) and publishes a 1st Go message if that is necessary. The services should remain unaware of this database, but the resolvers (i.e. BRE) could use this database, for instance to determine if a certain message is the last message in the batch. Could include creating BRE vocabularies.
Add support for request-response.
Add a DynamicItineraryService that will allow to adjust/override an itinerary.
Support absolute DateTimes rather than TimeSpans for the Batch and Message Timeouts.
Support xpath properties on XxxPipelineServiceResolution to support assigning of batch id and sequence id using xpaths (rather than the BatchId and SequenceId promoted properties, defined in ESB.Extensions,Schemas).
Add per-instance pipeline configuration to the configuration settings for the pipeline services, and rewrite the xlangpipelinemanager to be able to accept the per-instance config.

Supported scenarios
The following scenarios should be supported:
1. Message debatching: See BizUnit testcase T00100.
2. Debatching and Aggregation, including Message and Batch timeouts: See BizUnit testcase T00200 - T00240.
3. Debatching, Resequencing and Aggregation: See BizUnit testcase T00300 - T00310.
4. Debatching, Resequencing and Aggregation whilst tracking batches using the EsbBatchDb: See BizUnit testcase T00350.

The ESB.Extensions was facilitated by Sela Technology Center http://www.sela.co.il/.

Commented Issue: GoPublisherDisassembler Property Promotion problem [8963]

BVeldhoen — Mon, 23 Nov 2009 16:37:17 GMT

I know you're not supposed to create issues on stuff that you wrote yourself :), but I don't know a way around this one, so I'm hoping someone else will....

The GoPublisherDisassembler pipeline component derives from XmlDisAsm, and has all the same configuration properties as XmlDisAsm. Next to those, it has two other config properties:
* ResolutionBeforeDisassembling (bool): if true, the resolution will be performed before disassembling, and the input message (only one!) is provided to the resolution provider. If false, resolution will be performed after disassembling, for each disassembled message, and the disassembled message is provided to the resolution provider every time.
* ResolutionConnectionString: The resolver config string specifying which resolution provider to use.

The GoPublisherDisassembler works when ResolutionBeforeDisassembling == true. It also works when ResolutionBeforeDisassembling == false, but only if the promoted property values, that are specified on the schema corresponding to the disassembled messages, are not used in the resolution process.

Example: The ESB.Extensions.Tests.BizTalkArtifacts project uses the SequenceId property, which is a promoted property on Order. The OrderBatch is disassembled into the separate Order messages. When the GetNext() function of the GoPublisherDisassembler pipeline component executes, this property has not been promoted yet. I guess that this property is promoted by the XmlDasmReader when the stream is consumed, which is later than when the resolution is executed (which happens in the GetNext function). My resolution depends on this promoted property, so I'm kinda stuck on this one.

Any solutions out there?

Updated Wiki: Home

BVeldhoen — Mon, 23 Nov 2009 16:34:45 GMT

ReceivePipelineService: Can be used to execute a receive pipeline from within the orchestration. One obvious application for this service is (post-receive) debatching using the XmlReceive pipeline (or any custom pipeline that includes the XmlDisassembler or FFDisassembler). This service can be configured using the ESB.Extensions resolution framework with the following parameters:
- Receive pipeline type
SendPipelineService: Can be used to execute a send pipeline from within the orchestration. One application for this service is aggregation using pipelines, that contain the XmlAssembler or FFAssembler components. This service, too, can be configured using the ESB.Extensions resolution framework with the following parameters:
- Send pipeline type
- Batch Timeout: The timeout to apply to the entire batch. When this timeout expires, all the collected messages up to that point in time will be aggregated and the service completes.
- Message Timeout: The timeout to apply to each message. When this timeout expires, all the collected messages up to that point in time will be aggregated and the service completes.
- XLANGMessageComparer type: The SendPipelineService uses a serializable list to collect the messages to be aggregated. You can specify the comparer type to control how the messages are sorted. This happens in memory, but does not add much overhead compared to the use of Microsoft.XLANGs.Pipeline.SendPipelineInputMessages.
Resequencer and ResequencerGo services: These two services can be used in conjunction to resequence a given sequence (or batch) of messages.

Fix the Itineraries in the designer. They are outdated compared to the exported itineraries in xml, which were manually edited.
Add a pipeline component that registers an incoming batch (in a database? EsbBatchDb?) and publishes a 1st Go message if that is necessary. The services should remain unaware of this database, but the resolvers (i.e. BRE) could use this database, for instance to determine if a certain message is the last message in the batch. Could include creating BRE vocabularies.
Add support for request-response.
Add a DynamicItineraryService that will allow to adjust/override an itinerary.
Support absolute DateTimes rather than TimeSpans for the Batch and Message Timeouts.
Support xpath properties on XxxPipelineServiceResolution to support assigning of batch id and sequence id using xpaths (rather than the BatchId and SequenceId promoted properties, defined in ESB.Extensions,Schemas).
Add per-instance pipeline configuration to the configuration settings for the pipeline services, and rewrite the xlangpipelinemanager to be able to accept the per-instance config.

Supported scenarios
The following scenarios should be supported:
1. Message debatching: See BizUnit testcase T00100.
2. Debatching and Aggregation, including Message and Batch timeouts: See BizUnit testcase T00200 - T00240.
3. Debatching, Resequencing and Aggregation: See BizUnit testcase T00300 - T00310.
4. Debatching, Resequencing and Aggregation whilst tracking batches using the EsbBatchDb: See BizUnit testcase T00350.

*) The failure of test T00050 is a known issue; see issue #8963 on the codeplex issue tracker.

The ESB.Extensions was facilitated by Sela Technology Center http://www.sela.co.il/.

Updated Wiki: Home

BVeldhoen — Mon, 23 Nov 2009 11:10:15 GMT

ReceivePipelineService: Can be used to execute a receive pipeline from within the orchestration. One obvious application for this service is (post-receive) debatching using the XmlReceive pipeline (or any custom pipeline that includes the XmlDisassembler or FFDisassembler). This service can be configured using the ESB.Extensions resolution framework with the following parameters:
- Receive pipeline type
SendPipelineService: Can be used to execute a send pipeline from within the orchestration. One application for this service is aggregation using pipelines, that contain the XmlAssembler or FFAssembler components. This service, too, can be configured using the ESB.Extensions resolution framework with the following parameters:
- Send pipeline type
- Batch Timeout: The timeout to apply to the entire batch. When this timeout expires, all the collected messages up to that point in time will be aggregated and the service completes.
- Message Timeout: The timeout to apply to each message. When this timeout expires, all the collected messages up to that point in time will be aggregated and the service completes.
- XLANGMessageComparer type: The SendPipelineService uses a serializable list to collect the messages to be aggregated. You can specify the comparer type to control how the messages are sorted. This happens in memory, but does not add much overhead compared to the use of Microsoft.XLANGs.Pipeline.SendPipelineInputMessages.
Resequencer and ResequencerGo services: These two services can be used in conjunction to resequence a given sequence (or batch) of messages.

Fix the Itineraries in the designer. They are outdated compared to the exported itineraries in xml, which were manually edited.
Add a pipeline component that registers an incoming batch (in a database? EsbBatchDb?) and publishes a 1st Go message if that is necessary. The services should remain unaware of this database, but the resolvers (i.e. BRE) could use this database, for instance to determine if a certain message is the last message in the batch. Could include creating BRE vocabularies.
Add support for request-response.
Add a DynamicItineraryService that will allow to adjust/override an itinerary.
Support absolute DateTimes rather than TimeSpans for the Batch and Message Timeouts.
Support xpath properties on XxxPipelineServiceResolution to support assigning of batch id and sequence id using xpaths (rather than the BatchId and SequenceId promoted properties, defined in ESB.Extensions,Schemas).
Add per-instance pipeline configuration to the configuration settings for the pipeline services, and rewrite the xlangpipelinemanager to be able to accept the per-instance config.

Updated Wiki: Home

BVeldhoen — Wed, 18 Nov 2009 20:53:44 GMT

ReceivePipelineService: Can be used to execute a receive pipeline from within the orchestration. One obvious application for this service is (post-receive) debatching using the XmlReceive pipeline (or any custom pipeline that includes the XmlDisassembler or FFDisassembler). This service can be configured using the ESB.Extensions resolution framework with the following parameters:
- Receive pipeline type
SendPipelineService: Can be used to execute a send pipeline from within the orchestration. One application for this service is aggregation using pipelines, that contain the XmlAssembler or FFAssembler components. This service, too, can be configured using the ESB.Extensions resolution framework with the following parameters:
- Send pipeline type
- Batch Timeout: The timeout to apply to the entire batch. When this timeout expires, all the collected messages up to that point in time will be aggregated and the service completes.
- Message Timeout: The timeout to apply to each message. When this timeout expires, all the collected messages up to that point in time will be aggregated and the service completes.
- XLANGMessageComparer type: The SendPipelineService uses a serializable list to collect the messages to be aggregated. You can specify the comparer type to control how the messages are sorted. This happens in memory, but does not add much overhead compared to the use of Microsoft.XLANGs.Pipeline.SendPipelineInputMessages.
Resequencer and ResequencerGo services: These two services can be used in conjunction to resequence a given sequence (or batch) of messages.

Fix the Itineraries in the designer. They are outdated compared to the exported itineraries in xml, which were manually edited.
Add a pipeline component that registers an incoming batch (in a database? EsbBatchDb?) and publishes a 1st Go message if that is necessary. The services should remain unaware of this database, but the resolvers (i.e. BRE) could use this database, for instance to determine if a certain message is the last message in the batch. Could include creating BRE vocabularies.
Add support for request-response.
Add a DynamicItineraryService that will allow to adjust/override an itinerary.
Support absolute DateTimes rather than TimeSpans for the Batch and Message Timeouts.
Support xpath properties on XxxPipelineServiceResolution to support assigning of batch id and sequence id using xpaths (rather than the BatchId and SequenceId promoted properties, defined in ESB.Extensions,Schemas).
Add per-instance pipeline configuration to the configuration settings for the pipeline services, and rewrite the xlangpipelinemanager to be able to accept the per-instance config.

Updated Wiki: Documentation

BVeldhoen — Wed, 18 Nov 2009 20:38:07 GMT

More detailed information can be found on my blog about the ESB.Extensions:
http://geekswithblogs.net/BVeldhoen/archive/2009/09/11/debatching-aggregation-and-resequencing-using-the-esb.extensions.aspx
http://geekswithblogs.net/BVeldhoen/archive/2009/08/28/my-beef-with-the-esb-toolkit-2.0-resolution-framework.aspx

Source code checked in, #46860

BVeldhoen — Wed, 18 Nov 2009 20:37:04 GMT

[NEW RELEASE] - Corresponds to 1.2.0

Created Release: ESB.Extensions 1.2.0 (Nov 18, 2009)

bveldhoen — Wed, 18 Nov 2009 20:26:01 GMT

See the InstallationInstructions-1.2.0.txt

Released: ESB.Extensions 1.2.0 (Nov 18, 2009)

Wed, 18 Nov 2009 20:26:01 GMT

See the InstallationInstructions-1.2.0.txt

Source code checked in, #46850

BVeldhoen — Wed, 18 Nov 2009 20:02:31 GMT

[TESTED] - Added EsbBatchDb functionality.

Updated Wiki: Home

BVeldhoen — Wed, 18 Nov 2009 19:49:04 GMT

ReceivePipelineService: Can be used to execute a receive pipeline from within the orchestration. One obvious application for this service is (post-receive) debatching using the XmlReceive pipeline (or any custom pipeline that includes the XmlDisassembler or FFDisassembler). This service can be configured using the ESB.Extensions resolution framework with the following parameters:
- Receive pipeline type
SendPipelineService: Can be used to execute a send pipeline from within the orchestration. One application for this service is aggregation using pipelines, that contain the XmlAssembler or FFAssembler components. This service, too, can be configured using the ESB.Extensions resolution framework with the following parameters:
- Send pipeline type
- Batch Timeout: The timeout to apply to the entire batch. When this timeout expires, all the collected messages up to that point in time will be aggregated and the service completes.
- Message Timeout: The timeout to apply to each message. When this timeout expires, all the collected messages up to that point in time will be aggregated and the service completes.
- XLANGMessageComparer type: The SendPipelineService uses a serializable list to collect the messages to be aggregated. You can specify the comparer type to control how the messages are sorted. This happens in memory, but does not add much overhead compared to the use of Microsoft.XLANGs.Pipeline.SendPipelineInputMessages.
Resequencer and ResequencerGo services: These two services can be used in conjunction to resequence a given sequence (or batch) of messages.

Fix the Itineraries in the designer. They are outdated compared to the exported itineraries in xml, which were manually edited.
Add a pipeline component that registers an incoming batch (in a database? EsbBatchDb?) and publishes a 1st Go message if that is necessary. The services should remain unaware of this database, but the resolvers (i.e. BRE) could use this database, for instance to determine if a certain message is the last message in the batch. Could include creating BRE vocabularies.
Add support for request-response.
Add a DynamicItineraryService that will allow to adjust/override an itinerary.
Support absolute DateTimes rather than TimeSpans for the Batch and Message Timeouts.
Support xpath properties on XxxPipelineServiceResolution to support assigning of batch id and sequence id using xpaths (rather than the BatchId and SequenceId promoted properties, defined in ESB.Extensions,Schemas).
Add per-instance pipeline configuration to the configuration settings for the pipeline services, and rewrite the xlangpipelinemanager to be able to accept the per-instance config.

Updated Wiki: Home

BVeldhoen — Wed, 18 Nov 2009 19:45:35 GMT

ReceivePipelineService: Can be used to execute a receive pipeline from within the orchestration. One obvious application for this service is (post-receive) debatching using the XmlReceive pipeline (or any custom pipeline that includes the XmlDisassembler or FFDisassembler). This service can be configured using the ESB.Extensions resolution framework with the following parameters:
- Receive pipeline type
SendPipelineService: Can be used to execute a send pipeline from within the orchestration. One application for this service is aggregation using pipelines, that contain the XmlAssembler or FFAssembler components. This service, too, can be configured using the ESB.Extensions resolution framework with the following parameters:
- Send pipeline type
- Batch Timeout: The timeout to apply to the entire batch. When this timeout expires, all the collected messages up to that point in time will be aggregated and the service completes.
- Message Timeout: The timeout to apply to each message. When this timeout expires, all the collected messages up to that point in time will be aggregated and the service completes.
- XLANGMessageComparer type: The SendPipelineService uses a serializable list to collect the messages to be aggregated. You can specify the comparer type to control how the messages are sorted. This happens in memory, but does not add much overhead compared to the use of Microsoft.XLANGs.Pipeline.SendPipelineInputMessages.
Resequencer and ResequencerGo services: These two services can be used in conjunction to resequence a given sequence (or batch) of messages.

EsbBatchDb
The EsbBatchDb can be used to track batches, so that batches can be dynamically combined or otherwise defined based upon the resolver implementations.

TODO
A number of issues are still open, if anyone's interested to take on any number of them:

Fix the Itineraries in the designer. They are outdated compared to the exported itineraries in xml, which were manually edited.
Add a pipeline component that registers an incoming batch (in a database? EsbBatchDb?) and publishes a 1st Go message if that is necessary. The services should remain unaware of this database, but the resolvers (i.e. BRE) could use this database, for instance to determine if a certain message is the last message in the batch. Could include creating BRE vocabularies.
Add support for request-response.
Add a DynamicItineraryService that will allow to adjust/override an itinerary.
Support absolute DateTimes rather than TimeSpans for the Batch and Message Timeouts.
Support xpath properties on XxxPipelineServiceResolution to support assigning of batch id and sequence id using xpaths (rather than the BatchId and SequenceId promoted properties, defined in ESB.Extensions,Schemas).
Add per-instance pipeline configuration to the configuration settings for the pipeline services, and rewrite the xlangpipelinemanager to be able to accept the per-instance config.

Updated Release: ESB.Extensions 1.1.0 (Sep 17, 2009)

bveldhoen — Mon, 09 Nov 2009 22:22:15 GMT

Contains the same basic functionality as 1.0.0, but now with some internal improvements and refactorings.

The .msi's are based on the BizTalk Deployment Framework 5.0, so you need to install that as well: http://biztalkdeployment.codeplex.com/. I had to make some adjustments to the .targets file regarding Busines Rules.

Pre-installation steps:

Install the ESB Toolkit 2.0 (and all it's prerequisites, off course).
Install the BizTalk Deployment Framework.
Copy the BizTalkDeploymentFramework.targets to the C:\Program Files\MSBuild\DeploymentFrameworkForBizTalk\5.0 folder.

Installation

Install the ESB.Extensions.msi (it's a regular Windows installer, don't import it into BizTalk, just double click it).
Run Deploy ESB.Extensions from the Windows start menu.
Install ESB.Extensions.Tests.
Manually edit the file C:\Program Files\ESB.Extensions.Tests\Deployment\EnvironmentSettings\SettingsFileGenerator.xml, so that the file locations point to the right ones.
Run Deploy ESB.Extensions.Tests from the windows start menu.
If you didn't edit the SettingsFileGenerator.xml: Create the appropriate file locations. The ESB.Extensions.Tests BizTalk application contains file locatons pointing to C:\Z\DEV\0\..., so you'll need to create them or change the file location configuration in the ESB.Extensions.Tests application.
You can use the OrderBatch and Go files from the ESB.Extensions.Tests source code to test the scenarios. (Sorry, you'll have to download the sources as well).

Post-installation steps:

Copy the esb.config to the C:\Program Files\Microsoft BizTalk ESB Toolkit 2.0 folder (and run the ESB Configuration tool again to update the configuration in the SSO database if you're using the SSO database instead of File location).

Released: ESB.Extensions 1.1.0 (Sep 17, 2009)

Mon, 09 Nov 2009 22:22:15 GMT

Install the ESB Toolkit 2.0 (and all it's prerequisites, off course).
Install the BizTalk Deployment Framework.
Copy the BizTalkDeploymentFramework.targets to the C:\Program Files\MSBuild\DeploymentFrameworkForBizTalk\5.0 folder.

Installation

Install the ESB.Extensions.msi (it's a regular Windows installer, don't import it into BizTalk, just double click it).
Run Deploy ESB.Extensions from the Windows start menu.
Install ESB.Extensions.Tests.
Manually edit the file C:\Program Files\ESB.Extensions.Tests\Deployment\EnvironmentSettings\SettingsFileGenerator.xml, so that the file locations point to the right ones.
Run Deploy ESB.Extensions.Tests from the windows start menu.
If you didn't edit the SettingsFileGenerator.xml: Create the appropriate file locations. The ESB.Extensions.Tests BizTalk application contains file locatons pointing to C:\Z\DEV\0\..., so you'll need to create them or change the file location configuration in the ESB.Extensions.Tests application.
You can use the OrderBatch and Go files from the ESB.Extensions.Tests source code to test the scenarios. (Sorry, you'll have to download the sources as well).

Post-installation steps:

Copy the esb.config to the C:\Program Files\Microsoft BizTalk ESB Toolkit 2.0 folder (and run the ESB Configuration tool again to update the configuration in the SSO database if you're using the SSO database instead of File location).

Source code checked in, #46184

BVeldhoen — Mon, 09 Nov 2009 22:18:24 GMT

[TESTED] - Previous functionality/unit tests still working. - Still need to implement, test (and fix) the EsbBatchDb, notification and policies.

Source code checked in, #45462

BVeldhoen — Sat, 24 Oct 2009 13:46:04 GMT

[COMPILES] [UNTESTED] - EsbBatchDb + BatchComplete notification - Still needs the resolvers (BRE rules) + unit tests

Source code checked in, #45184

BVeldhoen — Sun, 18 Oct 2009 10:35:55 GMT

[UNTESTED] - Halfway implementing EsbBatchDb.

Source code checked in, #45180

BVeldhoen — Sun, 18 Oct 2009 09:21:21 GMT

Source code checked in, #45179

BVeldhoen — Sun, 18 Oct 2009 09:20:29 GMT

Source code checked in, #45178

BVeldhoen — Sun, 18 Oct 2009 09:19:37 GMT