MailСhimp IBM Integration Bus and MicroSoft Dynamics CRM
Integration with .NET node
From 8 versions WMB supports integration with MS .NET using new .NETCompute and .NETInput nodes. In addition, two new patterns are available for MicroSoft Dynamics CRM integration.We are going to provide the sample of complex integration between SAAS application mailchimp.com and MS Dynamics CRM. In addition the usage of Java Compute Node is discussed.
MailChimp is an email marketing service provider, founded in 2001. It has 7 million users that collectively send over 10 billion emails through the service each month. Mailchimp provides auto sigh in functionality that allows getting new leads trough well created advertized landing pages.
Microsoft Dynamics CRM is a customer relationship management software package developed by Microsoft. Out of the box, the product focuses mainly on Sales, Marketing, and Service (help desk) sectors, but Microsoft has been marketing Dynamics CRM as an XRM platform and has been encouraging partners to use its proprietary (.NET based) framework to customize it.
One of the recent trend in lead generation recently became creating landing pages optimized for google search, adding subscription form to it and advertize them trough paid channels, for ex. Google Adwords.
One of the things I love about Dynamics CRM is how easy it is to create data interfaces to enable integration with other systems. If you’ve worked with Dynamics CRM for any length of time, you’ve probably seen multiple web service integrations that enable interoperability with other line- of-business and legacy systems. A typical pair of inbound and outbound integrations might use outbound web service call to update destination application and receive inbound call via Dynamics CRM web service.
Using a tightly coupled connection between the source and target systems is usually the easiest (thus the quickest and cheapest) way to establish an integration, but this is often a bad idea. Consider the inbound scenario in which an external application is sending data to Dynamics CRM. What happens if the calling application misbehaves and starts sending thousands of requests per second? This has the potential to overload your CRM server and make it completely unusable. Now consider the outbound scenario in which a CRM plug-in calls an external web service. If the destination application’s web service is offline for a few minutes, the update from the CRM plug-in will not get received unless there’s some sort of error handling and retry logic built into the plug-in
An alternate approach
For these reasons, and lots of others (logging, security, scalability, just to name a few), it’s considered a best practice to create loosely coupled integrations that rely on a message broker that sits between the source and destination systems. Though the formal definition is more complicated,
for our purposes a message broker can be thought of as a collection of queues that hold messages. Publishers write messages to queues, and then consumers pick up the messages and process them appropriately. Additionally, the message broker can be configured to keep messages in their queues until the consumers provide confirmation of successful processing.
For the outbound call from the CRM plug-in, the plug-in writes the message to a broker. The message is routed to a queue where it waits to be processed. A separate processing service application retrieves the message from the queue and sends it to the destination application. For the inbound call to CRM, the process works exactly the same, except the source and destination applications are reversed.
Why is a message broker better?
In the inbound call scenario, an effective message broker would typically be expected to handle a larger volume of inbound messages than Dynamics CRM because all it’s doing is receiving and routing the data without any additional processing. The processing service can then process the messages in the queue at a speed that doesn’t overload the Dynamics CRM server. In the case of the outbound call, the combination of a message broker and processing service can enable complex retry logic and custom logging without having to store it in the plugin layer. As an added bonus to either scenario, a message broker can provide a guarantee that messages don’t get lost between the source and destination systems as long as the message is successfully published to the broker.
Overall wiring diagram
As soon as the new prospect sign in to the mailing list it becomes interesting to propagate his coordinates into on premises CRM solution.
Landing page, signup form and webhook
First of all we create landing page with signup form attached to the new mailing list on mailchimp. As soon as it’s done and tested, we have to prepare the middleware.
Mailchimp.com provide functionality that allows calling backend with parameters of changes happened online via webhook. The webhook call is nothing but the regular HTTP POST with parameters that describes the nature of event and sends the data of changed object.
In order to create the webhook we log in to the mailchimp account hit the Lists menu, choose the list and go to the Settings->Webhooks-> Create new webhook
HTTP input handling and MailChimp to GBO mapping
We have to create a message flow to receive the webhook call and translate it to standard XML message.
Parameters for HTTP Input:
Basic->Path suffix for URL* : /mcmqrequest*
(as in the webhook URL, see above)
Input Message Parsing->Message domain: BLOB : For messages with an unspecified format
The code from JCN for POST request parsing :
// ---------------------------------------------------------- // Add user code below
// Retrieve InputRoot/BLOB/BLOB field: MbElement outerblob = inMessage.getRootElement().getLastChild(); MbElement blob = outerblob.getLastChild();
// Serialize to byte array byte mybytes=(byte) blob.getValue();
// Parse byte array to string with UTF 8 encoding String urlencoded = new String(mybytes, "UTF-8");
// Parse string with POST parameters to key value map Map asbosub = asMap(urlencoded); // Create Subscriber from JAXB generated POJO
Subscriber gbosub = new Subscriber();
// Populate standard mailchimp fields gbosub.setEmail((String)asbosub.get(URLEncoder.encode("data[email]", "utf8"))); gbosub.setFirstName((String)asbosub .get(URLEncoder.encode("data[merges] [FNAME]", "utf8"))); gbosub.setLastName((String)asbosub .get(URLEncoder.encode("data[merges] [LNAME]", "utf8"))); Object outMsgJavaObj = gbosub; // End of user Java object processing // ----------------------------------------------------------
We map fields from application specific business object (ASBO) asbosub that we create by parsing POST application/x-www-form-urlencoded data to generic business object (GBO) gbosub.
In order to generate the Subscriber class we create first the xml schema description of the GBO. The same xsd that we are going to use later for MS Dynamics CRM pattern.
After parsing the data below goes to the output queue :
<subscriber> <email>email@example.com</email> <first_name>Claudiuss</first_name> <last_name>Lachmann</last_name> </subscriber>
Instantiation of the .NET calling pattern
In order to query and modify data in the CRM backend we will use Microsoft Dynamics CRM Account Entity output: Dynamic transport input pattern
The configuration data looks like this :
Similar artifacts for delete update and retrieve will be generated.
Next step is to map the GBO to the ASBO of MS Dynamics CRM in the Create mapping node and the response retrevied during .NET assembly call in CreateReturn node.
For this sample we map only a few fields :
MSMCCRE.IN MSMCCRE.OUT MSMCRET.IN MSMCUPD.IN MSMCDEL.IN MSMCCRE.OUT MSMCRET.OUT MSMCUPD.OUT MSMCDEL.OUT MSMCLOG MSMCERROR
Then we test and deploy message flow on the broker IS :
Now as soon as the new user is added in the mailchimp form it will appear automatically in the MS Dynamics CRM :
If you want to create .NET dll for message broker nodes by yourself it is necessary to use MS Visual Studio 10 or 12 with .NET version 4.5. Unfortunately you will not succeed with version 13 of MS VS. With the Express Edition of MS VS the coding is fine but you will not be able to use symbolic debugger.