Did you miss part one of this blog post? Check it out here before continuing.
Citrix ADM 12.1 build 49.23
We began with the most recent firmware, Citrix Application Delivery Management (ADM) 12.1 build 49.23. In the release notes, we saw some improvements in the highly available code, which was meant to make Citrix ADM failover more reliable. Unfortunately, appliance failover in this version presented some bugs which led to some headaches.
For example, after initiating failover for the first time, convergence was greater than five minutes. If failover occurs a second time without more than a half an hour delay, it can take 10+ hours to complete. During this time, we saw heartbeat failure alerts in the event logs and other odd errors. Simply put, the high availability fixes in the release notes did not function as intended. This issue was present in Citrix ADM appliances deployed in both client’s environments, and after testing it in my home lab using the same build. Citrix also confirmed that other customers were reporting the same issue. If you are configuring highly available appliances, I recommend not using this build.
Citrix released Citrix ADM 12.1 build 49.37 after the project began. Ultimately, we settled on this version which resolved the failover issues as outlined above.
Citrix Application Delivery Management 12.1 build 48.18
Following our adventures in the Citrix ADM 12.1 build 49.23 firmware, we downgraded to 12.1 build 48.18. The Citrix documentation on downgrading is not clear. But after conversations with Citrix, I discovered that it should work as long as the build is in the 12.1 family. And, after testing this in my lab, the failover did complete without issue.
One cool feature of upgrading or downgrading the Citrix ADM appliances — you only need to initiate it on the primary node. Citrix ADM automatically replicates the build and upgrades the second appliance.
Missed features and issues with custom StyleBooks
By downgrading, we missed out on some feature enhancements from the newest firmware, yet things were stable. Unfortunately, we quickly ran into another issue. We were creating custom StyleBooks to streamline and produce more consistent vserver deployments on the Citrix ADCs. When running the StyleBook, numerous JSON errors occurred. An issue also emerged with sorting items through the Citrix ADM GUI, including Events we did not see in the Citrix ADM 12.1 build 49.23 firmware. Another issued caused monthly logs not to sort properly in ascending order, making it more difficult to find logs from older timeframes.
Finally, logging into the Citrix ADM GUI as an external Active Directory user leveraging LDAP authentication, presents an error when running any StyleBook (see above). The error does not appear if you log into Citrix ADM as the local “nsroot.” After troubleshooting, we found that a known bug caused this issue, and was easily replicated in my home lab.
BUG0716793 StyleBooks | MAS API lookup fails with external user login. The issue is visible only when external users login.
Citrix ADM 12.1 build 50.28 will fix this bug. However, at the time of deployment this build was not available. If you require the StyleBook feature and have external users authenticating to Citrix ADM, do not use Citrix ADC firmware 12.1 48.18.
Citrix Application Delivery Management 12.1 “Secure Access Only” Setting
The Secure Access Only system setting in Citrix ADM disables access the GUI over the HTTP protocol and disables integration with Citrix Director over HTTP. Due to the client’s security requirements, we enabled this feature as per the design. When enabling Secure Access Only when running StyleBooks, you get certificate errors when using an external Active Directory account to log in. Local accounts like nsroot do not experience the same issue.
As per Citrix support, Citrix ADM 12.1 build 50.28 will fix this issue. If your security requirements force you to disable HTTP access to the Citrix ADC appliances, you will need to upgrade to the Citrix ADM 12.1 build 50.28+ to resolve this issue. Or, only deploy StyleBooks using nsroot or another local superuser account.
Citrix Application Delivery Management 12.1 Incompatible with Newer Versions of VMware NSX
This client also wished to integrate Citrix ADM with VMware NSX. However, in a newer version of VMware NSX, version 6.3, the integration is no longer supported. After working with Citrix, we found that VMware NSX version 6.3 altered the API commands that Citrix ADM uses for service insertion. In fact, VMware no longer supports third party service insertion, which changes up the approach with which third party load balancers are leveraged in an NSX environment.
At this time, VMware recommends deploying and using vRealize Automation with vRealize Orchestrator to automate the deployment of NSX network services and Citrix ADC load balancing.
Citrix Application Delivery Management 12.1 Bug Creating ECV Monitors with StyleBooks
The last bug we found when creating Extended Content Verification monitors through StyleBooks. We use these monitor types when it is not enough to see that a TCP connection is successful. Instead, we send a type of request to the backend servers that looks for a response to validate the monitor, such as a GET string to a specified site or page. Read more about custom ECV monitors here.
With Citrix ADM, if you configure a custom ECV type monitor through a StyleBook, the Citrix ADM server does not format the request to create the monitor correctly when sending to the Nitro API on Citrix ADC instances. Since the formatting is wrong, the ADC rejects it. As such, there is no way to use custom ECV monitors. Citrix is planning to fix the support mentioned this issue in Citrix ADM 12.1 build 50.28. Alternatively you can create a ConfigJob using Record and Play, or an ADC CLI command to create the monitor from the Citrix ADM appliance. Read more about using ConfigJobs with Citrix ADM here.
Conclusion
Since working on the project, two firmware releases have been made available from Citrix. Citrix ADM 12.1 50.30 seems to be the best bet, though there are still a couple of lingering issues. The heartbeat and failover problems and sorting issues are now resolved, the system seems stable and fast, and several other issues are fixed as well. The updates and resolved issues can be seen in the Citrix ADM 12.1 build 50.30 release notes. If any of the lingering issues are relevant to you, be sure to check the Citrix ADM release notes in future builds to see if they are resolved.
The intent of this blog is not to be negative about the Citrix Application Delivery Management tool. On the contrary, my experience with Citrix ADM leads me to believe it should be a staple in any Citrix networking deployment. By leveraging Citrix ADM, you gain better visibility into your traffic and real-time status on the health of all your Citrix ADC instances. Further, you benefit from easier central management of SSL certificates and licenses. You also receive methods for the central configuration of your instances and vservers that provide a more consistently configured, and therefore a more secure, Citrix networking environment.
I am excited about this product. It is relatively new, and combines the features of Citrix Insight Center, Command Center, and Control Center in one valuable tool. Citrix is currently developing features and fixing issues very quickly. However, even in its current state I recommend deploying and starting to learn the features. I think you will be impressed and it will change the way you manage the traffic using Citrix networking for the better.
