You just went Live and Dynamics AX Performance is crawling. First impressions are everything and this is one early impression that no one wants. This isn’t as uncommon as you would believe. Performance tuning new implementations is the most frequent request that I get at this time of year, and I’d like to share some observations of what it takes to get past this situation.
Performance Survival Point 1: Performance Tuning is like Getting or Staying in Shape
So often, I’ll be talking to people and hear them get angry at somebody because performance is slow. Realize that when it comes to Dynamics AX performance, the system acts just like the human body. Just as different levels of fitness require different amounts of effort and time to achieve or maintain, so does Dynamics AX performance. Just as the level of fitness is dependent on the person (6 pack versus a size 38 waist), the level of performance is dependent on the company. The long end of it is that you have to continuously stay at it and actually work on it with a plan of action.
Performance Survival Point 2: Functional Configuration mismatching is usually the biggest cause of Day 1 Go Live problems
I’ve seen all sorts of things cause Day 1 go live problems, but the most common cause is mismatching functional configurations. The configurations can mismatch with ISV’s, partner code, Microsoft code, hardware, etc. For example, one time, I saw a setting on an inventory dimension cause AX to scan each inventory record 54,000 times before picking it. The ISV code had a conflict with a new dimension that was added. Or, I’ve seen company configurations cause all kinds of problems with general ledgers. When it comes to functional configurations, they are easiest to fix at the beginning before there is a lot of data.
Performance Survival Point 3: Get an Assessment
There are some very good performance assessments around. Microsoft offers their version of one that is almost automated. Several other people, including myself, offer one. The biggest takeaway from the assessment is walk away with a plan of action and then execute it. Performance tuning won’t happen without a plan. Hardware, software configurations, loads, and various other conditions need to be tested. None of us started assessments for our health. They came from experience and learning what works through trial and error.
Or my assessment and there are others who offer one. Each one is a little different. The point is to get an assessment.
Performance Survival Point 4: Trained Employees are the only Money Saving Solution in the long run
That’s me at a billion dollar organization performance success story, celebrating a last day with some knowledge transfer. I’ve been on small implementations that had 5 people using Dynamics, and I’ve been on multi-continent, mega-implementations with thousands of users. The one rule that has stayed the same is that companies who made it had employees who eventually acquired the skillset to handle day to day Dynamics AX performance. Performance work never goes away. You’ll either spend way more money paying someone with constant headaches or have employees managing the new challenges as they come along. As a benchmark, I’d say the goal should be to get the employees to being able to handle 90% of the performance issues within 6 months, leaving on the niche stuff for consultants (crashes due to strange errors, custom data archiving, configuration mismatch, etc).
Performance Survival Point 5: Everything must be proven with Real Numbers
While some things are subjective, the first thing I always tell an organization that I’m mentoring is that everything must be proven with real numbers. If real quantifiable metrics aren’t able to be produced than something is wrong with the detection methods that you are using for Dynamics AX Performance issues. If you are using someone who can’t detect the condition than get someone else who can do it. This is critical. Always capture a before tuning load in a case where the users are complaining and an after tuning load while performance tuning. We tune until we get the quantifiable running load to match the organization needs.
As an example, take a look at one of the ways that I show how to quantify Dynamics AX performance:
Performance Survival Point 6: If in the Azure cloud, highly consider investing in a Performance Tuning VM for troubleshooting that is as close in Hardware to Production as Possible
Lifecycle Services Telemetry covers less than 5% of what a good, experienced performance tuner can see. In the long run, nothing is as efficient as educated employees with knowledge of the business processes carrying out performance. However, to do this, they need the tools, which are locked down in the Production Cloud. Therefore, highly consider getting a duplicate of production. It will probably save you more than all the money you will spend performance tuning and troubleshooting.
Performance Survival Point 7: Beware of “Juicing” a beginning implementation for a supposed quick win
With even a little bit of experience, a beginning implementation is very easy for someone to “juice up” with bunches and bunches of indexes off the suggested index view and call this a win. There are other quick win/non-sustainable strategies also. This will temporarily increase performance to a certain extent, but solely following this strategy usually leads to a big crash and loss of confidence later on. Thankfully, Microsoft is very good about detecting over-indexing and other “juicing” strategies when they occur. Indexes play an important role in performance tuning but there is a way to do it right. Much of your work should be making existing indexes more efficient or changing configurations to go with them if possible. While initially a little longer, the long-terms benefits are over-whelming. There are times where the priority is to just get things going, but these should always be temporary.
Performance Survival Point 8: Never give away a big contract without proven results in the contract
I talk to a lot of companies who are out of budget by the time that they approach me, often going through multiple places until they are out of budget. This should never happen. Write a results based contract with quantifiable numbers for payment. If you don’t know what that is than talk to someone who does, so they can explain it to you. I’m considered very good at performance tuning, but I know others who are also good. None of us have a problem with a results based contract because we aren’t used to our performance tuning projects going bad. Look for that so that you don’t waste your money and budget.
Performance Survival Point 9: Don’t expect your performance tuner to give away everything
Performance tuners are like custom chefs. Most things with good cooking are about following uniform best practices but there is a small percentage that makes all the difference in the world between skill level. Within performance tuning, 85% of it is the same. This is what I teach to the employees. 15% of it is entirely dependent on the individual and undocumented. I’m so serious about my performance tuning secrets that I wouldn’t share them with my own wife. They have taken countless years and hours upon hours of unpaid late night study to master and learn, and they allow me to stay in business year over year. However, I have taught many organizations how to handle Dynamics AX performance and handle it well. The goal should be to fix the issues and become sufficient in handling them on your own. At this stage of my career, I’m proud to say that some of the organizations that I have taught now have their own 15% of highly niche, undocumented secrets. It’s about the fundamentals first.
Performance Survival Point 10: It is okay to prioritize some things for later
I’ve been on several successful assignments, but I’ve never been on one where we achieved every single thing that we wanted to achieve. Sometimes, there are certain constraints that only allow you to go so far (usually money). That’s okay. I haven’t had abs with a 6 pack in many years, but I have gotten in shape from time to time. You get it. Just like a regular implementation, there is always a needs versus wants analysis and a budget factor that ultimately causes some things to be left out.
In Summary, this is what I tell every Dynamics AX Performance implementation before I start out. I hope that it helps some of you out there during this time of go-lives. This is a good ERP system once you get to enjoy it.