On-Premise upgrade from Dynamics 365 v.8.2 to v9.0

In my team, we want to stay close to cutting edge, but not bleeding edge. We won’t be first mover on the newest technology, but as soon as the teething problems have been worked out, and the risk has been assessed, we are ready to go. That is why we decided that after v.9 had been out for a year, we should upgrade to the latest version. Of course, we would prefer to go to CRM Online, but for different reasons, we are not quite ready for that journey yet.

This post describes the way we planned and executed the upgrade, and what we learned from it. The content is not only applicable for v.9, but a lot of it is relevant for any upgrade project.

Planning for the v.9 upgrade

When upgrading CRM, you have a few options:

• In place upgrade: Run the installer on your existing servers
• Migration upgrade: Install the new version on new servers, and import/upgrade a backup of the database
• Data migration: Rebuild CRM on new servers, and migrate the data.

In general it is always recommended to do a migration upgrade over an in-place upgrade. The reason is related to risk. If an in-place upgrade screws up your environment, you’ll have to re-install your servers and restore databases to latest backups. If a migration upgrade goes wrong, you can just delete the organization and try again. There is also the added benefit of being able to do dry runs of the upgrade, and you have a solid fall back option: You just keep your DNS pointing to the old servers.

The data migration option makes sense if you have a lot of technical debt in your solution, and a lot of bad data. This is especially the case in many companies where an initial CRM implementation has been made, but they have not done the maintenance needed to keep the system up to date. When you are 4+ major versions behind, you can be sure to face a lot of breaking changes, and code that needs to be rewritten. It is in such cases I would consider starting over, and migrating the data that is valuable to you, and use the migration as an opportunity to do some data quality clean-up.

Luckily, in our case, we have a great focus on following best practices and coding features in a way that is forward- and Cloud-compatible. Therefore there was no reason to start from scratch, and the migration upgrade was the obvious choice.

Also, the old CRM was on legacy servers, where the Bank has a strategy to move to private cloud solution (in-house) for hosting. We saw this as an opportunity to not only upgrade CRM, but to redesign the system architecture into something that would give us more control, flexibility, security and scalability.

Managing scope

Upgrading Dynamics 365 can be a big task on its own, but redesigning the system architecture at the same time, on a new platform with new technologies makes this a very complex project. It became clear very early that we needed to reduce scope as much as possible.

We managed to reduce the scope of the upgrade by doing the following:

Classic UI

Most of the sexy features in Dynamics 365 is in the Unified Client Interface. However, going directly from v.8.2 to UCI would require a lot of change management and organizational implementation, and the effort would have to be done in one go up to Go-Live.

By instead going to Classic UI, we can drastically reduce the training effort, as users are able to adapt to the improved classic UI without additional training. Once upgraded, we can gradually pilot and roll out the UCI at our own pace.

Therefore, by choosing Classic UI, we reduced the upgrade project from being a project impacting the entire team at a large scale, to a technical project to be delivered by a task force.

ADFS and Outlook

ADFS is the recommended mechanism for authentication in Dynamics 365. Yet, we decided not to implement ADFS in the initial phase for the following reasons:

• We don’t have a lot of in depth experience configuring ADFS in the team
• Setting up will require a lot of cooperation with other teams, causing external dependencies and risk of delays
• Kerberos/NTLM is the fall-back for ADFS, so this has to be configured anyway.
• Connections strings for ADFS are slightly different, so by delaying the switch to ADFS, the upgraded deployment will be easier to get to work with existing integrations
• The Outlook client for v.8 is compatible with v.9, so by staying on Kerberos authentication, we don’t have to do anything about the client configuration for our many users.

SDK

Testing concluded that CRM v.8.2 SDK was compatible with Dynamics 365 v.9. Likewise, we tested that we could deploy components built towards CRM v.9 SDK to an Dynamics 365 v.8.2 environment.

This gave us the option of updating the code before upgrading CRM, upgrade the code and CRM together, or wait with upgrading the SDK until after the CRM upgrade.

We decided to wait with upgrading the SDK until after going live. However, certain tools like the solution packager needed to be upgraded to v.9 in order to work with the new environments.

Once we did upgrade the SDK, it was not trivial, because you cannot mix and match versions. Once we upgraded one part of the code, we needed to upgrade our entire codebase.

Branching and code-freeze

The challenge for us was to be able to work in a way that we could prepare the infrastructure for V.9 while developing new features for v.8.2 in parallel. However, at one point we would have to upgrade our development environment, and at this point, we would not be able to release any more features for v.8.2 until the Go-Live date for the upgrade (except for minor hot-fixes). The trick is therefore to switch over to developing on v.9 as close to the go-live as possible.

By reducing scope of the project, and having a strict focus on pre-loading tasks, we were able to reduce the time from switching development track to V.9, and to going live, down to less than two months.

By planning the upgrade like this, and doing the work in a small task force, we were able to continuously deliver value to the business while working on the upgrade in parallel. In the end we only had to skip one planned release (we release every three weeks), and by doing it this way, the upgrade was a lot more acceptable for the business owners.

So how is a migration upgrade actually done? Here is the overview of the phases once the infrastructure is in place:

1. Test upgrade
   1. Try to upgrade a backup of prod onto the new servers
   2. Note down all issues, and make plans for how to cope
   3. Re-do test upgrade to confirm plan if needed
2. Switch branching and dev environments
   1. Once you are confident that you can upgrade the production environment over the course of a night or a weekend, you can proceed with this step
   2. Upgrade the DEV (and TEST) environment
   3. Update the develop and release (if you have that) branches in your source control repository to contain the v.9 code and customizations
   4. Keep at least one non-prod environment on v.8.2, along with one branch (ie. master) to reflect the current production, in case you need to do hotfixes.
3. Make all the necessary changes to your code and customization to be ready for go-live.
4. Go-live
   1. Do the actual upgrade by backing up the production DB, and restoring it in the new environment.
   2. Deploy all the changes that was prepared since switching to v.9 development.
   3. Update the DNS entries to point to the new infrastructure
   4. Pop the Champagne!
5. Fall-back?
   1. Just change the DNS back. But remember, once users get on the system, and start updating data, there is not turning back!

“v.9 Upgrade” is an infrastructure project

At least it was true in our case!

We decided to name the project “v.9 upgrade”. This was a good name, as it was easily understandable for both the business and the IT organization. “Private cloud migration” might have been more descriptive, but where is the value for the users in that?

From past upgrades I have been part of, we have had the infrastructure delivered by technical consultants or IT departments, so we could focus on upgrading the actual solution and code. Based on my previous experience, I estimated that we could do the upgrade in about three months.

However, building a complex infrastructure for an Enterprise scale CRM deployment takes time. A challenge not to be underestimated is the strict segregation of duties and specialization implemented in a huge enterprise organization. This makes a lot of sense for day to day maintenance, but when doing a big technical project, you quickly realize that being dependant on others for even the most trivial tasks can kill any progress.

In the end, we delivered the upgrade to v.9 in about 7 months, with a task force consisting of about only 1/4 of the team. The actual upgrade of the solution and code was done primarily in the last month.

Our Architecture

Our old architecture was based on virtual servers hosted in the LAN zone. We had 6 of them, and they were all deployed as full servers, containg CRM, custom API’s, custom integrations services etc. The database was deployed to physical servers for performance reasons.

In the new architecture, we have made the following improvements:

• Each environment separated from each other in different network Enclaves.
• Each server has only one function, with the following server roles:
  • CRM front-end
  • CRM back-end
  • Custom API’s
  • Integrations
  • Custom UI and Apps
• The different server roles are hosted in separate network microsegments.
• Each server role is load balanced between instances in two datacentres.
• All communication in, out and between microsegments are controlled by firewall rules
• New servers are much smaller than the old ones. However, there are a lot more of them, and they can be scaled up and down as needed via the cloud portal. This makes the servers more resilient (they impact each other less), and we maintain a reasonable cost level while being able to scale up drastically if needed.

Terraform

One of the great benefits from switching to a cloud deployment was the outlook of doing infrastructure as code deployments. Our cloud solution supports terraform, and we had high hopes for using terraform, even though the implementation was fairly new.

However, terraform turned into a great source of frustration and delay for us. The main reason being that the terraform provider for our cloud solution was not mature enough at the time, and we ended up scrapping terraform after 1.5 months of trying to get it to work. The main issue was that the provider was really bad at handling changes, and would suggest to redeploy the entire infrastructure if you had changed a load balancer setting. That is not cool when you have spent weeks installing and configuring the servers.

I cannot judge terraform as a technology based on our experience on one project, using an in-house provider in a very early stage. However, I can see terraform being very useful if you need to spin up new environments with the same configuration often. If you have a complex infrastructure, and you will doing more changes than complete redeploys, I would at least be more careful before adopting the technology. It would be great in a disaster recovery scenario, to completely rebuild an environment, but you have to assess if the amount of work is worth it. And sometimes it is almost as easy to provision the components by hand when you have a cloud self service portal, and a good guide. For us, we might revisit terraform later, once other have had more success with it, but for now it is buried.

Issues upgrading to Dynamics 365 v.9

Windows Server 2016 and SQL Server 2016

There are a few things you need to be aware of when installing Dynamics 365 on the 2016 version of Windows Server and SQL server:

• Dynamics 365 pre-requisites: You need to install the 2012 version of SQL Server Native Client, even though the other SQL assemblies should be the 2016 version (CLI tools and management objects).
• AES encryption: On Windows Server 2016, your installation user need to have AES encryption enabled in order to install CRM. Otherwise you will get the below error message. You will also need your domain admin to enable AES encryption on the service account for the CRM application pool, or Kerberos authentication will not work for CRM.

Kerberos

Kerberos requires some experience to set up, and luckily I have done it before. At the same time, we had the old v.8.2 environment where it was working, so it should be easy enough, right?

In reality, setting up Kerberos was a b**ch…

Kerberos is a very secure provider for Windows authentication. In IIS, Kerberos is the Windows Authentication provider called “Negotiate”. It allows single sign on in an intranet environment.

For Kerberos to work, you need to register SPNs (Service Principal Name) for the service account of the application pool. The SPN is registered in the Active Directory, and tells the AD which domain user is allowed to encrypt and decrypt Kerberos access tickets, for a given DNS name. It is not allowed to have duplicate SPN entries, so that means that any given DNS entry can only be backed by a single service account.

In a load balanced deployment, that means that all front end servers must use the same service account. At the same time you need to configure IIS to use the application pool credentials for Kerberos, and not the machine account. Otherwise each front end server will try to authenticate users using their own machine account, which would be different for each server, and not matching the SPN. You can do this in IIS by opening the Configuration Editor, navigating to system.webServer/security/authentication/windowsAuthentication and setting the parameter useAppPoolCredentials to true.

There is another setting in the same area called useKernelMode. Kernel mode is an easier way to set up Kerberos, where you don’t need to have SPNs. It is using the local machine account to do this, so it makes sense that this setting should be false when useAppPoolCredentials is true. However, after much trial and error, we found out that Kerberos would only work in our set up if both parameters were set to true.

The next issue we faced was that the load balancers in the new cloud environment were not the same as the ones in the old environments. Apparently, the default setting on AVI loadbalancers is to use connection multiplexing. Connection multiplexing is not a feature that plays well with Kerberos and NTLM. Unfortunately, this was not a setting we could just disable in the cloud portal, and we had to go through a long process including a very long waiting time, to get to a network admin that could do it for us (without even knowing if that would fix it for us).

The last issue we had with Kerberos is already mentioned above in the last section. Apparently Kerberos works different on Windows Server 2016 than on different Windows versions. For it to work, we needed to get someone to enable AES 128 bit encryption and AES 256 bit encryption on the service account of the CRM application pool.

The biggest issue I have with Kerberos is that it is impossible to debug. It either works, or it does not. If you have more than one thing not working in your setup, you can try endless combinations before getting anywere. My advice? Talk to an expert!

Permissions, permissions, permissions…

Because we wanted to potentially automate the installation of Dynamics 365, and also control what AD groups was used for CRM, we used an unattended command line installation, using a configuration file:

https://docs.microsoft.com/en-us/dynamics365/customerengagement/on-premises/deploy/use-command-prompt-install-dynamics-365-server

In order to control the AD groups created, you need to set the auto group assignment to false in the configuration file. This will however also prevent the installer from assigning all required privileges to the service accounts. We also had issues getting the installer to accept service credentials in the config file, so we ended up installing with default credentials (Network Service) and changing the service users in Services and IIS afterwards.

To get away with that, you need to be sure that you have assigned the correct privileges yourself. Luckily, Microsoft has documented the least amount of privileges needed:

https://docs.microsoft.com/en-us/dynamics365/customerengagement/on-premises/deploy/security-considerations-for-microsoft-dynamics-365

However, when we were done installing, we discovered quickly that following Microsoft’s recommendations were not enough to run Dynamics 365. We found the following issues with the recommended permissions:

• Application Service (IIS app pool user) needs to be granted log on as a batch on back-end servers, or the application will fail upon iisreset. The CRM website is not in use on back end servers, so we discovered this after a Windows patch that restarted IIS, and caused the CRM app pool user to be locked out of the AD!
• Sandbox processing service will not start with recommended permissions by MS. We ended up having to uninstall the sandbox service, and re-install it via the uninstall/configure wizard in program files. That way, we were able to re-install it from the UI, where we could type in the correct credentials, and have the wizard apply missing (and undocumented) permissions. We never found out what permissions were actually missing.
• Async service will run with documented permissions, but will generate errors in trace files and event viewer. By re-configuring the async service in the UI wizard (like with the sandbox service) we managed to get it working.

Manually assigning permissions is painstaking compared to just clicking next in an installer. However, I have found one interesting thing through this experience: On many deployments that I have seen before, there have been many warnings about failure to generate CRM keys in the event viewer. For example, the old environment on v.8.2 was not able to generate keys in the config database unless the async service was restarted every night. However, after properly assigning permissions (UI installer + manual assignments) to the async and async maintenance services, these errors have disappeared in the new environments.

Patches

So which patch should you apply when upgrading?

We have experienced some serious issues with different versions of Dynamics 365 v.9.

With version 9.0 (installation files), it is not possible to create new organizations or import old organizations. Apparently it is only good for in-place upgrades.

We managed to successfully upgrade from v.8.2 by applying update 0.6 (9.0.6) to our servers. This was a fairly stable release, but during final testing, we found some severe bugs that were blocking for us going live. For example, there were several views, dashboards and charts that did not render properly. Especially dashboards looked like crap, with dashboard elements jumping around randomly, creating misalignment and whitespace.

As a last resort fix, we applied update 0.12 (9.0.12) just before go-live. That fixed a lot of the UI issues. However, when doing test upgrades, you should always do it on the version you plan to go live on. We learned that the hard way during the Go-live weekend when it turned out that version 9.0.12 was not to happy about importing old databases much like version 9.0. Luckily we was able to work around it by temorarily downgrading a single back-end server to version 9.0.6, and then proceeding with the deployment.

Based on the issues we have had with the different patches, it is clear that Microsoft is not putting it’s strongest focus into the on premise version of CRM.

Testing and bugs

Microsoft always tells us to stop coding and use configuration instead. That way you are guaranteed to do things supported, and things will not break when upgrading.

Well, we have quite a lot of custom code in our deployment, but I am quite proud of the quality we put into our code, and so I was not that concerned about breaking changes. Also, we went for the Classic UI instead of Unified Client Interface, so it was pretty much v.8.2 with some new CSS, right??

Even so, we did quite a lot of testing, primarily focussed on our custom code, and integrations. We were so naive…

The initial finding revealed that there were not big issues, like expected. However, during the final weeks before Go-live, when the whole team had been moved over to v.9 in the development environment, we started getting all sorts of bug reports.

To our astonishment, more or less all the bugs was not at all related to our code, but bugs in the platform!

Examples of error we found on Dynamics 365 9.0.6 during the final two weeks before go-live:

• A view will not render if you have added customer JS to it (any JS), and one of the columns is an activity party.
  • Workaround: Remove either JavaScript or PartyList column.
• The field pipeline phase on opportunity will be calculated with incorrect number prefix
  • Workaround: Replace the field with a custom field, controlled by plugin and one-time job.
• CRM throws and error when you try to resolve a Case and there is a RetrieveMultiple plugin registered on ActivityPointer, that access the InputParameters on the plugin execution context.
  • Reported to Microsoft. Should be fixed by version 9.0.13
• CRM throws “An error occurred while the chart was rendering.” when creating a chart with a horizontal label that points to a field from Owning User and the view contains a criteria on Activity Parties.
  • Reported to Microsoft. They have promised that this will be fixed by 9.0.14, to be released in March.

Learnings regarding testing:

• Just because Microsoft calls something a facelifted UI, does not mean you should assume that it is working the same way as the previous version, and required less testing than UCI.
• Just because you do something “best practice” does not mean it cannot break during an upgrade. That is true for custom code, and that is apparently also true for something as simple as a view.
• Don’t underestimate testing, even though it is tempting as a tech person, when you are in a hurry, and running out of time.
• What is true in one version may not be true in another. When you install a patch, you cannot trust any previous test results. Test over.
• Don’t test your custom code (alone). Test what impacts the users and customers.

Thanks to a huge team effort in the final weeks, we managed to go-live without any big issues, and make v.9 a success!

Nordic Plum Fizz Cocktail [off-topic]

What is the best way to relax after a long days work? Personally, I enjoy good food and drink in company of good friends.

Over the years I have experimented a bit about making drinks at home, and it has resulted in a recipe I deem worthy of sharing. The key ingredient is my fathers home made preserve plums in sugar syrup, so it is a very exclusive drink 🙂

The plums he has in his garden are Victoria plums, but I am sure any other plum will work as well.

The drink is basically a sour, and I call it the Nordic Plum Fizz!

Ingredients

• 1,5 oz OP Anderson Aquavit
• 3/4 oz lemon juice
• 1/2 oz simple syrup
• 1/2 oz syrup from preserved plums
• 1/2 egg white (optional)
• A few drops of Angostura Bitter

Shake all above ingredients with ice.

Add one preserved plum in the bottom of the glass for decoration, along with Ice cubes.

Top with soda

Ingredients

How to get Unified UI in Dynamics 365 for Sales (Web/desktop)

With version 9.0 of Dynamics 365 (July 2017 release), a lot of new features has been unveiled. One of them is a brand new UI rendering engine, called Unified UI.

According to our impressions from the preview information, you’ll have the choice between the following form types:

• Dynamics 365 Enterprise Edition Web Forms – Old desktop forms, Refreshed with new look and feel
• Unified UI forms – For Business edition, mobile devices and Unified Service Desk

The UUI was for many one the key selling point of the long awaited Dynamics 365 Business Edition, based on a much nicer menu system and very nice looking forms. However, now it seams like Business Edition is “dead”, at least in its original form. What does this mean for users who can’t wait to use the new “Business Edition” UI in browser based access to CRM? – It turns out, it is already here…

After playing a bit around with trials on Dynamics 365 V9.0, I have become a bit wiser on the capabilities and consequences of the new UI system. I’ll try to summarise what I have learned here:

• Unified UI is not a Business Edition thing. It is available in Enterprise Edition, but just not the default UI.
• UUI forms are not forms of a new form type. It is just a new form renderer for existing forms.
• Whether to show the existing Enterprise Edition (Refreshed) web UI, or the new Unified UI, is controlled by the Dynamics 365 App system.
• There is no new form editor for UUI forms. You will need to use the old “CRM 2011” form designer, and apply some abstract thinking, to imagine how it will look as a UUI form.
• Mobile forms (MoCa) are now gone, and replaced by the Unified UI. THe only available form types inside a solution is now:
  • Main form
  • Quick create form
  • Quick view form

Q: Why would Microsoft go through the trouble of upgrading the existing forms, when a brand new form system is already available? – A: Unified UI is very significant change to the look and feel of CRM. It is as big a change as when the CRM 2013 forms and menues was released to replace the CRM 2011 UI. I believe the refreshed UI has been introduced to ease users into the UUI. At least while they work on the last missing features in Unified Interface. It looks like a mix of the old forms and the UUI forms anyway!

Limitations of Unified Interface

• I cannot seem to find the Advanced Find!!??
• There does not seem to be possible to apply filters on columns directly from looking at a view.
• Unknown at this point: Microsoft is replacing a lot of the old aspx pages with Custom Control Framework components. I am not aware if all aspx pages have been re-implemented yes. If not, I will imagine that there could be issues with exposing certain entities / settings through the UUI sitemap, for example security roles. Maybe this is why Advanced Find is nowhere to be found?
• I am sure there are other limitations as well, since Microsoft is not officially recommending to use UUI in the Enterprise Edition for web. If we are lucky, Microsoft will fix these before we even notice.

 

To start using Unified UI, you need to do the following steps:

1. Ensure you are updated to Dynamics 365 V9.0. Depending on the availability, you might need to create a new trial.
2. Create a custom UUI App under Settings > Application > My Apps
   1. In the properties tab of the App designer, make sure to set the client type as “Unified Interface” instead of “Web”.
   2. Add the desired forms and other content, and publish.
3. Click in the tile for the new custom App to open it, and you are done! This will add an app id to the URL of your CRM

 

Picture 1: List of Apps. Apps for UUI is tagged with “Unified Interface”

 

Picture 2: UUI Apps can be opened in the browser, with the new menu system and the new form renderer!

BusinessRules bug?: “The given key was not present in the dictionary” when creating record in Dynamics 365 v.8.2

I got the following error when creating a Contact in a migration job:

Details: System.ServiceModel.FaultException`1[Microsoft.Xrm.Sdk.OrganizationServiceFault]: Unexpected exception from plug-in (Execute): Plugins.ContactOnCreate: Microsoft.Xrm.Sdk.SaveChangesException: An error occured while processing this request. (Fault Detail is equal to Microsoft.Xrm.Sdk.OrganizationServiceFault).

When digging in the exception details, i found the following message: “The given key was not present in the dictionary”.

Info about my set up: On the create message of the Contact entity, I have a plugin that creates a predefined set of related records of a custom entity. On the related entity I have a business rule that set the contact lookup to read only if it has a value.

Explanation

When creating a CRM record with a lookup to another record where the name field is not set, the create request will fail if there is a business rule on the record you are trying to create, where the following conditions are met:

• The business rules scope is set to object
• The business rule has a condition that checks if the lookupv that points to an unnamed related record, is set

This scenario can happen in an integration or migration scenario, where some records might be created without a name value.

Resolution

You can avoid the error by ensuring that:

• The primary attribute (the name field) is business required on all entities
• Avoid setting business rule scope to object, unless you have to
• Ensure that all integrations, migration jobs and plugins create new record with at least a dummy value in the primary attribute, if no value is available.

The Art of CRM Batch Jobs

The more CRM projects I do, the more fan I become of the scheduled batch jobs. They are not a silver bullet solution, but in many scenarios they have many benefits over more traditional code customization for CRM, like form scripting, plug-ins and workflows. Of course, the different ways of customizing a CRM or Dynamics 365 solution differs a lot on what you want to accomplish, and I see all of the above as important tools in any CRM dev’s tool chest.

You will also often run into cases where more than one of your tools will be valid solutions to your problems, and that is when you should stop, and consider the pros and cons of each tool available to you, rather than just selecting the hammer time after time. In situations like this, I lately find myself going for a batch job approach more and more often. Why that is I will try to explain in this post, but the key factors for me (if implemented right) are:

• Scalability
• Performance
• Robustness

The typical batch job scenario

Unlike plug-ins, batch jobs are typically used for data processing that are not event driven, involving a large amount of records to be processed in a short amount of time. These are typically integration scenarios, or recalculation jobs. Most often, the data will be transactional.

Another typical scenario is that some logic should be executed on a record on a certain date, or when a certain condition is met, without any changes being made on that record at that time. For example, creating a reminder task for you to remind you of each contact you have who’s birthday it is today.

Plugins/workflows vs. Batch Jobs

There are a couple of caveats when working with plug-ins and workflows in CRM. Both types of customizations are built to be able to add some processing logic to save events on single records in CRM. They are not meant to be used for long running calculations or integrations, affecting many records in the CRM database. If you try to do this, you may run into the following challenges:

Both plug-ins and workflows have a 2 min timeout

Now, 2 min should be more than enough for anyone, but just think about it. You might have a business critical application running a plugin that for some reason runs very slow, and suddenly it just stops! Well written plug-ins or workflows shouldn’t run more than a couple of seconds tops, and it you get longer execution times than that, you should consider if you made the right approach.

Plug-ins and workflows execute in transaction (DB-locking)

This means that if something goes wrong, the state of the affected records gets rolled back which is nice. However it also means that the SQL server will ensure that whatever data used as input for your logic, remains in the state you retrieved it, until the transaction is saved. This means that whichever records you retrieve within the plug-in/workflow, will be locked for read and write until the transaction is done. If you have plugins waiting for external resources (web service replies) running on multiple records, you very quickly get into deadlock scenarios which cause very bad performance.

NB: The read-lock can be bypassed, if you add the no-lock=’true’ hint to your fetchXml queries or QueryExpressions. This way you can read locked data, well knowing that the state of the data you read might be rolled back if a CRM transaction should fail. No-lock is not supported in Linq to CRM queries…

Waiting workflows as a scheduler

By using waiting workflows, it is possible to run some logic on a record at a specific date, or when a condition is met. However, in a transactional setting, waiting workflows are really bad as they fill up the asyncOperationBase table. CRM is not particularly efficient at handling a huge amount of unfinished workflows, and so this is something you should try to avoid.

Transactions in Batch jobs

Ever since CRM 2015, it has been possible to execute multiple request against CRM in a single transaction, using the ExcecuteTransactionRequest. This is a powerful tool for making robust batch jobs. The biggest advantage is that you can retrieve all the data you want without locking any rows in the database, and still get the roll-back functionality when storing the changes in a very quick operation (short locking).

Just remember that if you really need to lock the data you fetch, until the transaction is finished, you need to use a plug-in or workflow. The CRM SDK does not offer any mean of making DB locks on reads.

Also, the ExcecuteTransactionRequest allows you to stack multiple requests together in one request, but does not enable you to insert logic on the result of on request, before moving to the next. This is typically a drawback when creating linked records, and you need the id of the first created record before you can link nr. 2 to it. The workaround is to assign the id’s in code, before calling create. If you want to do this, you should generate sequential guids using this guide. Otherwise, CRM will not be able to sort the records in the created order without an explicit ordering applied. Be aware that the generation of the sequential guids use the servers MAC address, and since the CRM’s internal guids are created on the SQL server, the guids you generate will in most cases not be in sequence with records created manually in CRM.

Best practices for developing batch jobs

Re-use your batch logic for user requests

I prefer to use batch jobs to process data in bulk on a schedule, but at the same time allow a user to trigger the same logic on individual records manually. This can help leverage the disadvantage of scheduled jobs, that data is updated with a significate delay (ie. Hourly or nightly). I typically do this by implementing the core of the processing logic in a workflow assembly, that is then referenced from the batch job application. The logic that is being run on an individual record, can then be exposed to a CRM user as a synchronous ad-hoc workflow or a custom action. Both are implemented in practically the same way, and can be run from a command bar (ribbon) button. However, if you want to make a button to trigger the logic, I prefer to use a custom action, as you would not have to hard code the workflow id in JavaScript.

Cache non-transactional data

Caching is really efficient in batch jobs, because by nature of the job, many of the records being processed by the job will require the same input information. Therefore, caching should be applied for all base and configuration data used by the job. I use MemoryCache.Default in .Net, and for large configuration tables I cache individual entries, while for smaller tables (less than 200 records?), I often cache the entire dataset. Just make sure not to cache data that changes on a regular basis.

Transactional data (salesorders, opportunities) should never be cached, as it changes all the time, and the amount of data you would end up caching would be too much. In most cases, you would not get a performance benefit for caching transactional data anyways.

NB: If you are re-using your processing logic for custom workflow activities, remember that custom types needs to be serialized before caching (See earlier post).

NBB: MemoryCache is not working in CRM Online plugins and workflows, due to plugins running in isolation, and the sandbox process is being restarted every few seconds! If you have exposed you logic to ad hoc user requests in a CRM Online environment, your code will not fail, but you should not expect running workflows to benefit from caching. I you are in an online scenario, you should make sure not to fetch more data than necessary instead.

Use pagination for retrieving data

Make sure that your bach job logic is scalable, by making sure the job won’t crash with timeouts or OutOfMemoryExceptions. Do this by using paging in all large queries, and make sure that the processing of each row is independent of the other rows in the retrieved page set. Also, for the paging to make sense, don’t keep the retrieved data in memory after you are done processing it.

Save the processing state to the record itself, and split your processing into several independent stages

Rather than creating processing jobs that are long and complex with many dependencies that can make the entire processing fail, try to split the entire process up into smaller steps that are independent of each other. To accomplish this, you should update the records you are working on with the current processing status/stage along the way, so the processing can continue in another run, without starting over from scratch. In order to truly be able to re-process any individual record from any state/stage, you need to write code where processing of one record is completely independent from other records, even if the records are related by nature. This is really useful when using pagination to retrieve the record set to work on, because you can never be sure that all the records related to each other is retrieved in the same page.

When the data you are working on is naturally related, and you cannot relate them in-memory because you have not loaded all the records in memory, you need to maintain the relationships on the database level. This is not as efficient as loading everything into memory, and doing all the processing in one go, but is scales a lot better. The approach also results in more robust solutions, as you can see the intermediate state of records that failed processing, rather than rolling a processing back to scratch. Having records with detailed error codes related to specific processing stages, makes error handling much easier. The chances that a user will actually look at an error and try to fix it (data errors), also increases drastically if the error message is inside CRM, rather than buried in a log file.

Consider an example:

Let’s say we have a invoicing batch job responsible for creating invoices and sending them to the ERP system.

The initial process, which is run daily, could be like the following:

1. Query all salesorderdetail (orderlines), created the last 24 hours
2. Create invoice with invoice lines for each salesorderdetail (orderlines)
   • The invoices are bundled by the related account
3. When the invoices are created, they are sent to ERP through a web service, in a single call containing a list of invoices

The problems with this approach is the following:

• If one salesorderdetail fails processing, nothing can be invoiced on that account.
• If the processing logic is not built to expect errors, invoices will be left stranded in an error state, or an entire invoice might to be rolled back in case of errors (if saved in same transaction)
• If the queried dataset has grown very large, the job might fail with timeout or OutOfMemoryExeptions.
  • Alternatively, you can use paging to avoid this, but then you will not end up with a single invoice per account, as individual lines on the same account might be fetched in separate pages

To improve on the initial design, we can do the following:

1. Step – Create invoice lines:
   • Query all salesorderdetail (orderlines) in a specific state (i.e. No related invoice line, or invoice status == null)
     • Use paging to reduce the amount of memory used
   • For each salesorderdetail:
     • Create an invoice line
     • Do validation to ensure the record can and should be processed in the next step
     • Set the status on the invoice line (and salesorderdetail). This could also be a validation error code.
2. Step – Bundling:
   • Query all invoice lines that are validated OK
     • Use paging to reduce the amount of memory used
   • Try to find an draft invoice in CRM
     • If one is found, add the invoice line to it
     • If not create a new invoice in draft state
3. Step – Calc sums and Validate
   • Query all invoices in draft state
     • Use paging to reduce the amount of memory used
   • For each invoice:
     • Retrieve all invoice lines
     • Sum up amounts
   • Do final validation
   • Update status on invoice from draft to “ready to send”, or set an error code.
4. Step – Send to ERP
   • Query all invoices in “ready to send”
   • Generate message and call ERP web service
   • Update status on all the invoices

Each of these steps can be run completely independently of each other. If an error occurs, or the processing stops, the processing can continue exactly where it stopped the next time. This is because the records in CRM contains the exact state of the processing, as opposed to keeping state in-memory.

By the way: These steps could be separate batch jobs running at different intervals, or they could be different stages in the same job. It is completely up to you. However, if you expose the job logic as workflows or custom actions to be called by a user manually, it would be a good idea to be able to manually force each step individually. That gives the most control for the user.

Retrieve first, and save last

If possible, retrieve all the un-cached data you need in as few queries as possible. Then process your logic in-memory. Finally, save all the changes using an ExcecuteTransactionRequest.

When retrieving the data for the job, see if you can get all the data you need by joining on the record set used as input for the processing. This will be the optimal solution in terms of performance. In general, when considering performance in batch jobs, you can ignore the in-memory processing and the initial fetch of records to work on. For any long running job, the only relevant performance factors are the fetches made while processing individual rows, along with saving the changes.

Thread safety

Based on the nature of the data you are processing, and the business criticality of the application, you might want to consider thread safety. What you want to avoid is having multiple simultaneous instances of the job running at the same time, processing the same data. This can especially be an issue if you use the strategy of fetching all the relevant data for the entire batch in one big query, and doing the rest of the processing in-memory. The processing can run over a long period of time, and without some locking mechanism, there can be a risk that the data changes without being reflected in the job, while it is running.

Even if you fetch the data for each individual record when it is being processed, and you check if the record still needs processing, there is a risk of double processing. This can happen if the processing is done in a web service layer that supports multiple simultaneous threads, and transaction data is being cached. What can happen in such a scenario, is that when you have two threads processing the same large dataset, the second thread will catch up on the first thread, because when all the required data is cached it can run much faster.

When threads are running in parallel I have seen a case where both threads check if the data should be processed, processes the data in both threads, and then updates the record’s status to prevent further processing. Imagine what this can do in an invoicing scenario!

However, in the most common cases, you don’t need thread safety in batch jobs, because you will have the job hosted in a way that only allows one concurrent thread. As long as a user doesn’t copy the job locally, or a developer decides to debug against production etc. 😉

If you think you need thread safety, I recommend two approaches:

• Record level locking: If you would like to be able to run processing in multiple treads and you need to make 100 % sure that you have the current version of each record, you need the running thread to lock the individual records to be processed. You can do this by having the job call a web service, that performs the actual processing. In the web service application you can use System.Caching.MemoryCache to keep a list of records that are being processed, and that should be locked for other threads. Use the lock feature of C# whenever updating the list of locked records to avoid timing issues.
• Global locking: If you don’t need the granularity above, you can implement a lock that makes sure that only one instance of the job can run at the same time, wherever it is being run from. Do this by creating a Lock entity in CRM, create a single record for each job type, and update this record to lock/unlock for processing. You might also want to have a new instance of the job unlock or take over an existing lock, as a failsafe, if the lock has been held for an abnormally long time (Previous job failed to release the lock).

Use impersonation

Plug-ins and workflows automatically use impersonation in the way that any create or update request will run in the context of the user that triggered the event.

When you move the logic out into a batch job (for performance reasons), you often want to have the same functionality. Therefore, consider impersonating the user set as owner of the main record used for processing input when making changes to that record. Impersonation is done by setting CallerId on the OrganizationServiceProxy object.

Deployment options

On-premise deployment

For an on-premise deployment, I usually go with a simple console application that is scheduled Windows Task Scheduler.

Some people tell me they prefer to implement the jobs as Windows Services because it should be more reliable and stable. However, it is not my experience that Windows services are more reliable than a Scheduled Task, so I prefer the Task Scheduler because it has so many scheduling options.

Consider the following when using the Task Scheduler:

• The task needs to be set to execute whether the owning user is logged in or not.
• If you need to run the task as a background task, set the owner to “SYSTEM”
  • NB: This will grant full system privileges to the task, so be careful. You also might not be able to open log files etc. generated on the file system.
  • You won’t see the console open when the task is running for background tasks
  • There might be a performance difference between running the task as a background or a foreground tasks. Server systems are typically configured to prioritize background tasks, while PC’s are configured to prioritize foreground tasks.
• Ensure proper error handling. If the job fails completely, it needs to close itself down. Otherwise it will block further executions later.

For more complex scenarios, you could use Windows Task Scheduler in cooperation with a WCF service layer or WebAPI hosted on IIS.

Cloud deployment

The on-premise deployment is usually not very convenient if you are using Dynamics 365 Online. Instead you could use Azure WebJobs, which gives the same functionality as Scheduled Tasks, in an easy interface.

If you need a more complex set up, using web services, you can host a service application with Azure App Service Web App. You could also have a look at Azure Logic Apps which is really good for orchestration in between.

CRM JS webresource best practice for ribbon Commands / Forms

Introduction

As a CRM developer, working with script web resources in forms and ribbons, I have learned a few lessons over the years, and I have seen a wide spread from best practice implementation to the complete unstructured mess.

What I often see when taking over an existing implementation, is the lack of namespacing in the JavaScript webresources. A flat function based code structure, where a myriad of functions are clashed directly into the DOM, is not only hard to read, but increases the chance of overriding existing functionality. For example, some commonly seen function names are often “onLoad”, “onSave”, “validate” or “calculate”. Imagine it you are using two different web resources with conflicting definitions of “onLoad” on the same form, without namespacing; You are headed into trouble!

Another common “mistake” is using the same webresource for form events, and for ribbon commands. This is not a problem in itself, but you should be aware that the scripts will be loaded twice, and that it might make debugging more confusing. While form web resources are loaded as normal included scripts during form initialization, command code for ribbon buttons are loaded as unnamed dynamic script blocks when the ribbon loads. This means that you cannot set a breakpoint in the ribbon code before clicking a button the first time (because the script is not yet loaded). It also means that if you use the same web resource for the form and the ribbon, the ribbon dynamic script will overload the form script, and thus you will loose the ability to debug the loaded form script (The dynamic script block can still be debugged if you can locate it…).

This post is an attempt to share some best practices that can prevent these issues, based on my experience. Code examples are found in the bottom.

General JavaScript development

Switch to TypeScript

Do your coding in TypeScript to have better intellisense and compile time error checking. ReSharper 9+ have TypeScript support, to help you code even better TypeScripts

NB: To really be flying with TypeScripts, you should use third party libraries that have TypeScript definition files. Luckily there are Typescript Definition files for jQuery, AngularJS, XrmServiceToolkit, Xrm.Page API, etc. See more at http://definitelytyped.org/

Follow general best practices

Follow general best practices for JavaScript that also applies to TypeScript to some extent: http://www.w3schools.com/js/js_best_practices.asp

There are also many online resources to improve your TypeScript style and code quality. Read up on the official documentation and guides like https://github.com/panuhorsmalahti/typescript-style-guide

Here are also two good CRM specific links, from JoeCRM:

CRM JavaScript Best Practices

JavaScript Best Practices for Microsoft Dynamics CRM 2015

Handle dependency conflicts

Be aware that CRM and certain managed solutions uses common third party libraries like jQuery. If you are using different versions of these libs, you might run into problems were something become not compatible as the last loaded version overloads the others. jQuery has a noConflict function that can be used to load several versions of jQuery at once, and this is facilitated by dependency injection frameworks like requireJS (http://requirejs.org/)

Optimize your scripts for good performance

If you have a lot of script files, and it is hurting load time, consider bundling and minifying scripts. This can be done with for example Gulp (http://gulpjs.com/)

General CRM Scripting

Namespaces

ALWAYS use namespaces in CRM JavaScript. Especially if you divide your logic in multiple scripts. It will make it a lot easier to maintain and read if you can see exactly what is being referenced. Also, you should introduce namespaces from the start, mostly because it is a pain to change all the references later on.

I prefer to use a hierarchical namespace structure, with the following naming convention:

[Orgname].CRM.[EntityLogicalName].[Area]

For example: Contoso.CRM.Account.Form and Contoso.CRM.Account.Ribbon

I also prefer to have a separate namespace for internal functions and variables, so that the only thing exposed through the main namespace is the functions registered as commands or event handlers in the forms

For example: Contoso.CRM.Account.Form._privateMembers

Prepare for re-use

Avoid having duplicate code in your Form JS and Ribbon JS. Place common code in a third library instead if possible, like Contoso.CRM.Account.Common.js. This will not only make the code re-usable, but also easier to maintain as changes will only have to me made in one place.

Use Virtual Folders for webresources

To make it easier to keep you scripts organized, group them in folders on your development computer, and make the name webresource name reflect that path. This is also a recommended practice from Microsoft (https://msdn.microsoft.com/en-us/library/gg309473.aspx)

For example: A file C:\ContosoCRM\Webresources\JavaScript\Account\AccountForm.js could be registered as “new_/Account/AccountForm.js”, with “new_” being the CRM publisher prefix.

Ribbons / Command bar

Use an editor

Use a ribbon editor like Ribbon Workbench (https://ribbonworkbench.uservoice.com/). There are others, but Ribbon Workbench is the one I have had best experiences with.

Load the actual logic in the form when appropriate

If the command/logic is only to be used on the form ribbon/command bar, consider having the actual logic loaded by the form itself, instead of as dynamic script in the ribbon. This will make debugging easier, and promotes reuse of the logic for form events. Also, making changes to web resources is a lot faster if you don’t have to change the command definition in the ribbon at the same time (i.e. changed function signature).

For commands that should be available on the Entity home page and in related views, all required scripts should be loaded by the ribbon itself.

Debugging Ribbon Commands

If you have trouble debugging the dynamic script blocks, you can try the “Breakpoints in Dynamic JavaScript” feature in Chrome. This is described along with other debugging approaches in this post: http://blogs.msdn.com/b/crm/archive/2015/11/29/debugging-custom-javascript-code-in-crm-using-browser-developer-tools.aspx

Loading dependencies in Ribbons / Command bars

If you need to load other JavaScripts in a ribbon command before executing your function (i.e. jQuery), load them by calling the function “isNaN”. The reason for this is that a javascript action in ribbon XML requires a webresource and a function name. isNaN() is part of the ecma javascript definition, and will therefore always be declared. It will also always execute without errors, and has a very little overhead.

Add a Keyboard Shortcut to your button

I think it is a good idea to add keyboard shortcuts for command bar buttons in forms. I have successfully registered keyboard shortcuts in the form’s onLoad event using this library: http://www.webreference.com/programming/javascript-keyboard-shortcuts/2.html
Add a shortcut hint as a tooltip on the button in the ribbon, so you won’t forget.

Code examples

Ribbon command script (JavaScript)

This example uses a minimal implementation, where the actual logic is in a separate file that is used both by the ribbon and form events.

var Contoso = Contoso || {};
Contoso.CRM = Contoso.CRM || {};
Contoso.CRM.Account = Contoso.CRM.Account || {};

Contoso.CRM.Account.Ribbon = {
    helloWorldCommand: function () {
        Contoso.CRM.Account.Common.helloWorldCommand();
    }
}

Ribbon command script (TypeScript)

This example uses a minimal implementation, where the actual logic is in a separate file that is used both by the ribbon and form events.

module Contoso.CRM.Account.Ribbon {
    export function helloWorldCommand() {
        Common.helloWorldCommand();
    }
}

Common code used by Ribbon and Form (JavaScript)

var Contoso = Contoso || {};
Contoso.CRM = Contoso.CRM || {};
Contoso.CRM.Account = Contoso.CRM.Account || {};

Contoso.CRM.Account.Common = {
    helloWorldCommand: function () {
        alert("Hello World!");
    }
}

Common code used by Ribbon and Form (TypeScript)

module Contoso.CRM.Account.Common {
    export function helloWorldCommand() {
        alert("Hello World!");
    }
}

Form script (JavaScript)

Adds a keyboard shortcut in the form, to the ribbon button.

Contoso.CRM.Account.Form = {
    _privateMembers: {
        initialize: function() {
            //...
        }
    },

    onLoad: function() {
        // Register shortcuts using http://www.webreference.com/programming/javascript-keyboard-shortcuts/2.html
        shortcut.add("Ctrl+H", Contoso.CRM.Account.Common.helloWorldCommand, { 'type': 'keydown', 'propagate': true, 'target': document });

        Contoso.CRM.Account.Form._privateMembers.initialize();
    }
}

Form script (TypeScript)

Adds a keyboard shortcut in the form, to the ribbon button.

module Contoso.CRM.Account.Form {
    
    // private members
    function initialize() {
        //...
    }

    // public/exported members
    export function onLoad() {
        // Register shortcuts using http://www.webreference.com/programming/javascript-keyboard-shortcuts/2.html
        shortcut.add("Ctrl+H", Contoso.CRM.Account.Common.helloWorldCommand, { 'type': 'keydown', 'propagate': true, 'target': document });

        initialize();
    }
}

Limitations when using OData in CRM to retrieve records

This post describes the limitations of the OData endpoint in CRM 2011, which should still be relevant for CRM 2013 and 2015. There is no reason to believe that the OData endpoint will be developed further as Microsoft is working on a brand new WebApi to replace both the current OData endpoint and the SOAP endpoint.

Please refer to this article for general information on the available options when Qerying CRM using OData: OData system query options using the OData endpoint

Re-blogged from http://blogs.msdn.com/b/crm/archive/2011/03/02/using-odata-retrieve-in-microsoft-dynamics-crm-2011.aspx:

With the release of Microsoft Dynamics CRM 2011, Microsoft have added a Windows Communication Foundation (WCF) data services (ODATA) endpoint. The endpoint facilitates CRUD operation on entities via scripts using Atom or Json format. This post mentions some of the considerations when using the endpoint, specifically around the use of retrieves.

First, the operations supported over this endpoint are limited to create, retrieve, update and delete. The REST philosophy does not support other operations and so we followed J. Microsoft did not implement others since the story around service operations was not fully developed in the current WCF data services offering.

The $format and $inlinecount operators are not supported. CRM’s OData endpoint only supports $filter, $select, $expand, $top, $skip, $orderby

Some of the restrictions when using the implemented operators are.

Operator	Restrictions
$expand	Max expansion 6
$top	Page size is fixed to max 50 records $top gives the total records returned across multiple pages
$skip	When using with distinct queries, we are limited to the total (skip + top) record size = 5000. In CRM the distinct queries does not use paging cookie are and so we are limited by the CRM platform limitation to the 5000 record.
$select	One level of navigation property selection is allowed I.e. …/AccountSet?$select=Name,PrimaryContactId,account_primary_contact …/AccountSet?$select=Name,PrimaryContactId,account_primary_ contact/LastName&$expand=account_primary_contact
$filter	Conditions on only one group of attributes are allowed. By a group of attribute I am referring to a set of conditions joined by And/Or clause. The attribute group may be on the root entity …/TaskSet?$expand=Contact_Tasks&$filter=Subject eq ‘test’ and Subject ne null (or) on the expanded entity. …/TaskSet?$expand=Contact_Tasks&$filter=Contact_Tasks/FirstName eq ‘123‘ Arithmetic, datetime and math operators are not supported Under string functions, Substringof, endswith and startswith are supported
$orderby	Order are only allowed on the root entity.
Navigation	Only one level of navigation is allowed in any direction of a relationship. The relationship could be 1:N, N:1, N:N

Working with the OrganizationServiceContext

When developing .Net code that is working with CRM data through the CRM Organization Service, we use the OrganizationServiceProxy class of the CRM SDK. Alternatively, we can use the OrganizationServiceContext class, which is generated by crmsvcutil.exe, along with early bound proxy classes for each entity in our CRM organization. The OrganizationServiceContext class builds on top of the OrganizationServiceProxy, and adds a lot of extra functionality like tracking changes, managing identities and relationships, and gives you access to the Microsoft Dynamics CRM LINQ provider.

See more information and examples here:

Use the OrganizationServiceContext class

In most typical cases I will use the OrganizationServiceContext class in order to use Linq to query data in CRM. Linq is by far my favorite method to query CRM for the following reasons:

• Queries can be written type safe and with 100% intellisense support. I.e. no strings in your code to specify attribute names. Resharper is also a big help wriiting Linq to CRM.
• Compact queries, either with SQL like syntax or Lambda expressions.
• Prettier and more readable code than FetchXML and QueryExpressions.
• It is possible to UnitTest your queries, by stubbing the data sets in the CrmServiceContext, using Microsoft Fakes
• A query can return more than one entity type at the same time, by constructing an anonymously typed container as return object.

Linq to CRM also have some limitations, for which you should use FetchXML or QueryExpressions instead. Most of all, it is missing outer joins and the “in” operator. Other limitations you should be avare of are described here:

Use LINQ to construct a Query

Another useful aspect of the OrganizationServiceContext is when modifying data in CRM. This is done by attaching objects (entities) to the service context, and submitting the changes using the SaveChanges method. However, you need to be aware that CRM records are by default automatically added to the context when performing Linq queries, so saving changes might have unforeseen consequences.

Before you start creating and updating object through the service context, I suggest reading this excellent post by Scott Durow:

Do you understand MergeOptions?

Comparison of OrganizationServiceProxy and OrganizationServiceContext

	OrganizationServiceProxy	OrganizationServiceContext
Bulk insert/update supported	No. All Update/Create/Excecute calls are handled as individual web service calls.	Yes. All changes in the service context are processed in a single web service call with SaveChanges. This also support relating objects to each other in the context, and have multiple inter-relationships created in the same service call.
When to use	When you need simple CRUD operations on a single or a few records. When you are using Linq and don’t understand merge options, or the differences when modifying objects with the two different approaches When you are on CRM 2011 rollup 11 or earlier.	When you need to modify or create many objects at the same time, you should use bulk update for improved performance. When you already tracking objects you have retrieved through the service context, you can reduce the code by just saving changes to the retrieved objects

Creation of objects

OrganizationService

var account = new Account
{
    Name = "Test account",
    Address1_City = "Copenhagen"
};
var accountId = service.Create(account);

NB: The id is returned

OrganizationServiceContext

using (var ctx = new CrmServiceContext(service))
{
    var account = new Account
    {
        AccountId = Guid.NewGuid(),
        Name = "Test account",
        Address1_City = "Copenhagen"
    };
    ctx.AddObject(account);

    ctx.SaveChanges();
}

NB: The id is not returned. Id’s should be assigned manually in code before saving if the id should be used later in the code. Alternatively; use ctx.AddLink() to have references between created objects automatically created in SaveChanges, whithout having to know the Ids beforehand:

using (var ctx = new CrmServiceContext(service))
{
    var account = new Account
    {
        Name = "Test account",
        Address1_City = "Copenhagen"
    };
    ctx.AddObject(account);

    var contact = new Contact
    {
        FirstName = "John",
        LastName = "Doe"
    };
    ctx.AddObject(contact);

    ctx.AddLink(contact, new Relationship("contact_customer_accounts"), account);

    ctx.SaveChanges(); // Creates two objects, and sets a reference between them
}

Update of objects

OrganizationService

var accountUpdate = new Account
{
    AccountId = accountId,
    Name = "Test Account"
};

service.Update(accountUpdate);

OrganizationServiceContext

using (var ctx = new CrmServiceContext(service))
{
    var accountUpdate = new Account
    {
        AccountId = accountId,
        Name = "Test Account"
    };
    ctx.Attach(accountUpdate); // The object is now tracked, and will be evaluated by SaveChanges()

    ctx.UpdateObject(accountUpdate); // Equivalent of OrganizationService.Update()

    ctx.SaveChanges(); // This performs the actual call against CRM 
}

As mentioned earlier, objects that are retrieved from CRM through the service context, are automatically tracked, based on the merge options in effect. This can lead to unexpected errors if you are not aware of the Automatic tracking of objects:

using (var ctx = new CrmServiceContext(service))
{
    var trackedAccount = ctx.AccountSet.First(a => a.AccountId.Value == accountId); // Automatically tracked in context

    var accountUpdate = new Account
    {
        AccountId = accountId,
        Name = "Test Account"
    };
    ctx.Attach(accountUpdate); // Error: System.InvalidOperationException: The context is already tracking a different 'account' entity with the same identity.
    ctx.UpdateObject(accountUpdate);           

    ctx.SaveChanges();
}

Automatically generate Crm Service Context through CRM Developer Tools

I love the CRM Developer Tools that come with the CRM 2013 SDK, even though it has its shortcomings. One of the areas where it is lacking, is the missing Crm Service Context when generating wrappers. To fix this, add a file named CrmSvcUtil.exe.config to the folder C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\Extensions\Microsoft\Dynamics CRM 2011 Developer Tools\1.0 The file should contain the following, which will ensure the service context is generated:

<?xml version="1.0"?>
<configuration>
    <appSettings>
        <add key="servicecontextname" value="CrmServiceContext"/>
    </appSettings>
</configuration>

InvalidCastException when using MemoryCache in Custom CRM Workflow Activity

I have been struggling with the following error all morning.

I have a custom workflow activity in CRM 2011, that generates a PDF summary from CRM records. To maintain a reasonable good performance and flexibility, I have based the layout on a XML web ressource, where the layout configurations and images are cached using MemoryCache.

The code runs perfectly in Unit Test, but when running the actual workflow, I get the following error:

Workflow paused due to error: Unhandled Exception: System.InvalidCastException: [A]System.Collections.Generic.List`1[Crm.Pdf.PdfGenerator+AttributeDescription] cannot be cast to [B]System.Collections.Generic.List`1[Crm.Pdf.PdfGenerator+AttributeDescription]. Type A originates from 'mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089' in the context 'LoadNeither' at location 'C:\Windows\Microsoft.Net\assembly\GAC_64\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll'. Type B originates from 'mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089' in the context 'LoadNeither' at location 'C:\Windows\Microsoft.Net\assembly\GAC_64\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll'.
...
...
at System.Activities.CodeActivity.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager)
at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)

Explanation

It turned out the error occurs when caching types that are defined in the workflow assembly. CRM workflows are versioned, in the sence that running workflows will not update their implementation even if the actual workflow assembly is updated through plugin registration Tool. In this way, each time a workflow is writing to the MemoryCache, a new instance of the workflow assembly is being used. This causes a problem when fetching data from MemoryCache, because in order to cast the data back to the original type, not only must it be the same type and assembly, but also the same instance of the assembly.

Resolution

In order to make caching work in this scenario, there are two options:

• Rewrite your caching code to store general types, for example by doing serialization
• Add the custom types that needs caching to a separate assembly that can be stored in the Global Assembly Cache

I found the first option most preferable in my specific case, from a maintenance and deployment perspective.