Sunday, September 25, 2016

Taking first steps to DevOps world - part 2

This is second part of my multi part blog post series about suggested first steps (from point of view) to DevOps world.

Link to part 1

Environments defined by code

Purpose of scripting is automate routine tasks and release time for more important things. Another benefit on scripting is that script(s) will do steps always on same way.

Issue in traditional scripts is that dependency and error handling and especially version control is complex because it is impossible to say which scripts are applied to which environments.
Fortunately there is new tools like PowerShell DSC (Desired State Configuration) which can be used to define target environment/configuration and which will automatically make sure that correct scripts will be run on correct order.

Third lessons learned is that: Use code to define target environment/configuration instead of traditional scripts when possible.

Virtual servers

Here I have very simple example how all define these needed 12 virtual servers configuration using very simple piece of code. This example is done on Hyper-V environment (using xHyper-V module) but there is also similar module available for VMware on here.
configuration TestAppVMsToHyperV
        [string[]]$NodeName = 'localhost',







        [String]$State = 'Off',


    Import-DscResource -module xHyper-V

    Node $NodeName
        xVhd DiffVhd
            Ensure          = 'Present'
            Name            = $($VMName + ".vhd")
            Path            = $Path
            ParentPath      = "D:\Templates\NanoIIS_v1\NanoIIStemplate.vhd"
            Generation      = "Vhd"
        xVMHyperV NewVM
            Ensure          = 'Present'
            Name            = $VMName
            VhdPath         = $($Path + "\" + $VMName + ".vhd")
            SwitchName      = $SwitchName
            State           = $State
            Path            = $Path
            Generation      = 1
            StartupMemory   = $StartupMemory
            MinimumMemory   = $MinimumMemory
            MaximumMemory   = $MaximumMemory
            ProcessorCount  = $ProcessorCount
            MACAddress      = ""
            Notes           = ""
            SecureBoot      = $False
            EnableGuestService = $True
            RestartIfNeeded = $true
            WaitForIP       = $WaitForIP 
            DependsOn       = '[xVhd]DiffVhd'

########## Dev environment servers ########## 
# StartupMemory/ MinimumMemory = 2 GB
# MaximumMemory = 4GB
TestAppVMsToHyperV -NodeName "localhost" -VMName "DevFE01" -SwitchName "DevFE" -State "Running" -Path "D:\ExampleAppVMs" -StartupMemory 2147483648 -MinimumMemory 2147483648 -MaximumMemory 4294967296 -ProcessorCount 2
TestAppVMsToHyperV -NodeName "localhost" -VMName "DevFE02" -SwitchName "DevFE" -State "Running" -Path "D:\ExampleAppVMs" -StartupMemory 2147483648 -MinimumMemory 2147483648 -MaximumMemory 4294967296 -ProcessorCount 2

TestAppVMsToHyperV -NodeName "localhost" -VMName "DevBL01" -SwitchName "DevBL" -State "Running" -Path "D:\ExampleAppVMs" -StartupMemory 2147483648 -MinimumMemory 2147483648 -MaximumMemory 4294967296 -ProcessorCount 2
TestAppVMsToHyperV -NodeName "localhost" -VMName "DevBL02" -SwitchName "DevBL" -State "Running" -Path "D:\ExampleAppVMs" -StartupMemory 2147483648 -MinimumMemory 2147483648 -MaximumMemory 4294967296 -ProcessorCount 2

########## Q&A environment servers ##########
TestAppVMsToHyperV -NodeName "localhost" -VMName "QAFE01" -SwitchName "QAFE" -State "Running" -Path "D:\ExampleAppVMs" -StartupMemory 2147483648 -MinimumMemory 2147483648 -MaximumMemory 4294967296 -ProcessorCount 2
TestAppVMsToHyperV -NodeName "localhost" -VMName "QAFE02" -SwitchName "QAFE" -State "Running" -Path "D:\ExampleAppVMs" -StartupMemory 2147483648 -MinimumMemory 2147483648 -MaximumMemory 4294967296 -ProcessorCount 2

TestAppVMsToHyperV -NodeName "localhost" -VMName "QABL01" -SwitchName "QABL" -State "Running" -Path "D:\ExampleAppVMs" -StartupMemory 2147483648 -MinimumMemory 2147483648 -MaximumMemory 4294967296 -ProcessorCount 2
TestAppVMsToHyperV -NodeName "localhost" -VMName "QABL02" -SwitchName "QABL" -State "Running" -Path "D:\ExampleAppVMs" -StartupMemory 2147483648 -MinimumMemory 2147483648 -MaximumMemory 4294967296 -ProcessorCount 2

########## Production environment servers ##########
TestAppVMsToHyperV -NodeName "localhost" -VMName "ProdFE01" -SwitchName "ProdFE" -State "Running" -Path "D:\ExampleAppVMs" -StartupMemory 2147483648 -MinimumMemory 2147483648 -MaximumMemory 4294967296 -ProcessorCount 2
TestAppVMsToHyperV -NodeName "localhost" -VMName "ProdFE02" -SwitchName "ProdFE" -State "Running" -Path "D:\ExampleAppVMs" -StartupMemory 2147483648 -MinimumMemory 2147483648 -MaximumMemory 4294967296 -ProcessorCount 2

TestAppVMsToHyperV -NodeName "localhost" -VMName "ProdBL01" -SwitchName "ProdBL" -State "Running" -Path "D:\ExampleAppVMs" -StartupMemory 2147483648 -MinimumMemory 2147483648 -MaximumMemory 4294967296 -ProcessorCount 2
TestAppVMsToHyperV -NodeName "localhost" -VMName "ProdBL02" -SwitchName "ProdBL" -State "Running" -Path "D:\ExampleAppVMs" -StartupMemory 2147483648 -MinimumMemory 2147483648 -MaximumMemory 4294967296 -ProcessorCount 2

That code creates these servers based on my Windows Server 2016 Nano template which I have created using this command:
New-NanoServerImage -Edition Standard -DeploymentType Guest -MediaPath E:\ -BasePath .\Base -TargetPath .\Nano\NanoIIStemplate.vhd -ComputerName NanoTemp -Packages Microsoft-NanoServer-DSC-Package,Microsoft-NanoServer-IIS-Package
More information about how to create Nano server templates you can find from here.

I have been working with automation for years and biggest issue what I have seen is that there is always someone who will go can do some unexpected configuration/upgrade/etc manually and that will break the automation.

Fourth lessons learned is that: Use Core or Nano servers instead of full GUI version of servers on your code defined environments. That will keep persons who do not know what they are doing out of from your servers ;)

When you have that environment define code ready you need apply it to virtualization platform. Here is simple example how to apply it locally on Hyper-V server:
Start-DscConfiguration -Path .\TestAppVMsToHyperV
That can be of course done also remotelly and on production you probably want use DSC pull server instead of pushing these configurations directly to servers.

This was second part of this blog post series. I will try post next one near future. Thanks for reading :)

Friday, September 23, 2016

Taking first steps to DevOps world - part 1

I have been saying already 1-2 years that DevOps world is direction where our company should go. Now (finally) management is also started talk that we should evaluate DevOps principles and how include them our daily operations.

That why it is good time to start multi part blog post (I'm not sure yet how many parts there will be) where I will describe my opinions which are correct steps on this "road".

On my work career I have been working first eight years with infrastructure and now two last years more together with application development. With that background I see that I have quite good opinion from "Dev" and "Ops" point of view.

I will also pick most critical "lessons learned" from my text and summarize them on last post for these who are too lazy/not interested to read whole story.

Example application

Best places to start following DevOps principles is of course when you are creating totally new application or doing re-building existing one from scratch.

I will present my example application from that point of view.

Needed environment

When we think about modern, scalable cloud application which is as simple as possible it would contain following server roles:
  • Frontend
  • Business logic
  • Backend

And logical picture of needed infrastructure would look something like this:

And like we all know minimum set of environments for that kind of application is:
  • Development environment
  • Quality assurance environment
  • Production environment

Using traditional approach it would be very big task for Ops team to get all these environments up and running so here is first place we can get benefits using DevOps principles.

First lessons learned is that: Create firewall rules and load balancer configs manually (because you only need create them once) and automate servers installations (because you need multiple servers with same settings, you need to be able to scale up/down your environment and it makes upgrades much easier if you can always install new servers with latest binaries instead of upgrading old ones).

Handling configurations

Everyone who have even tried to configure environment like this manually knows that needed work effort is huge because each server role can contains multiple application components and each or them can have multiple config -files.

Our packaging team has used lot of time to develop installers which can handle all these configurations and to be honest they are done very good and important work with it. That was especially important earlier when all applications was single tenant applications and most of the installations was on customers' in-house environments.

Now when our company focus is on Cloud business and all new/re-built applications are multi-tenant I see that we can (and actually even need) to use "shortcuts" on here to be able create and update environments on cost-effective way.

Fortunately Visual Studio contains this very nice feature which can be used to generate Web.config -files for different environments automatically.

Here is example pictures of it configured to use:

There is also this nice extension which will allow you to do samething for all other *.config -files too.

I can already see that someone will ask question "What if we still need provide applications for in-house customers?" so I will answer it already. Nice thing on this feature is that you still can have one configuration which creates *.config files for your packaging process and these packages can be used on in-house installations.

Purpose of this configuration is define Cloud environments on code which allows you to do application deployments/upgrades without care about configs (which is important step on your way to continuous deployments). Another and maybe even bigger benefit on this configuration is that dev, QA and production configurations are all documented on code, under version control and you can be sure that settings are 100% same on all these environments.

Second lessons learned is that: Use Visual Studio *.config -file transform feature to generate and document different environments *.config -files.

This was first part of this blog post series. I will try post next one near future. Thanks for reading :)

Thursday, September 15, 2016

Troubleshooting performance issues caused by performance monitoring tool


It have been a while after my last post because I'm now working mostly with applications and not some much with infrastructure anymore. Anyway, now I have nice real world troubleshooting story which I like to share with you.

Everyone who has even tried troubleshoot application performannce issues knows that it is very time consuming. That why we decided to acquire Application Performance Management/Monitoring tool to help with it.

After comparing our requirements to tools which are available we decided to start proof of concept project using Dynatrace APM which is provided us by their local partner company here in Finland so we was able to be sure that log data does not leave the country (which was business requirement).

Findings in PoC

Installation package issues

After testing Dynatrace Agent to couple of servers I noticed that there looks be these two bugs on installation MSI package:

  • On some servers IIS agent installation fails but I did not found any real reason to it.
  • On some servers MSI package fails to store IIS module registration to IIS configuration.
    • On this case MSI log even says that installation was done successfully.
Another issue was that there is also quite many manual configuration tasks needed after installation.

Because we are planning to deploy Dynatrace Agents to multiple servers and enabling/disabling them where needed I decided to create PowerShell scripts which can be used to install Dynatrace agents and enable/disable them when needed and include workarounds to these issues to these scripts (These scripts are btw available here.).

Performance issues caused by performance monitoring

First test results from production environment was very shocking. Reports shown for us that there was huge slowness on one of the environments.

When I investigated reasons to this issue I found that slowness mostly happened evening/night times when there was just couple of users on system.

Then on one evening when I was troubleshooting this issue I noticed that biggest delays was there immediately after iisreset and slowness was much worse than I had seen earlier on any other environment even when I was only user on system.

Because Dynatrace Agent was only difference between this and other environments which worked fine I decided to try remove it also from this environment and suprise suprise first load time after iisreset decreased to one tenth of earlier.

Situation was very interesting that tool which should help us to improve application perfomance actually caused them.

When we investigated this with Dynatrace support we got information that is it actually normal that instrumenting .NET profilers will slow down application start little bit even we did some tuning to configurations.

As solution to this we decided to configure IIS to keep application pools always running and preload them immediately after iisreset based on this guide:

That looked fixing issue so I included also that feature to my enable.ps1 -script.

Difficulties moving to production mode

After done lot of testings on test environment we assumed that we are finally ready to go production with this tool and we decided to enable monitoring agents again to couple of production environments.

Assuming was once wrong again. This time issue was that most of the application pools did not started at all after enabling Dynatrace agent. Logs showed that IIS agents started fine without any errors but .NET agents not even tried to start.

Hidden IIS feature

This issue seemed to be tricky and it took long time for me, my colleagues and Dynatrace partner company to investigate. In the end we noticed that when we removed all configurations done by script and did all configuration steps manually it suddenly started working and that happened on two different environment but not on third one even we did all steps same way than ealier.

After comparing working and not working installations I found that there is actually one hidden feature on IIS Management Console.
When you register module using IIS console it checks if binary is 32 or 64 bit version and automatically adds precondition that module will be only loaded by application pool which are running on same bitness. PowerShell cmdlet "New-WebGlobalModule" does not do that automatically so if you do not give -Precondition parameter (which is not mandatory) IIS will try load that module to both 32 and 64 applications. And actually you need give same precondition again when you run "Enable-WebGlobalModule" cmdlet.

Here you can see that precondition is not visible on IIS console:

But here you can see that precondition is still on IIS config file:

or it can be missing like we had and you cannot see that from IIS console and it caused application pools starting issue.

This is also now fixed to my scripts.

Whose fault it was?

When you use lot of time for troubleshooting and finally got problem fixed it is also interesting to use sometime to think that whose fault it was after all?

If we think all issues I have explained here then we can probably say that:

  • It is Dynatrace fault that their MSI package does not fail if installing IIS modules fails.
    • Or maybe it is Microsoft fault because their appcmd.exe tools is used inside of MSI?
    • Or maybe it is Microsoft fault that it is their AppFabric process which locks IIS config when this happens?
  • It is Microsoft fault that IIS module registration using IIS console and PowerShell works differently.
    • Or maybe it is Dynatrace fault that they don't say on their documentation that we should care about this but they are still using "/precondition" parameter inside of MSI package?
    • Or maybe it was my fault that I did not readed all Microsoft documentations that how IIS module registration should be done using PowerShell?

Anyway, it was Olli's problem and after all I was able to fix it :)