Category Archives: Automation

A ninja walks into a #TurboFest

Screen Shot 2016-07-26 at 2.49.56 PM

*Very late posting this, but better late than never!*

On Jun 15th I was fortunate enough to attend my first VMTurbo TurboFest. VMTurbo is a company I’ve personally been watching for a while, and have always been excited to see the direction they are taking, not just in the Datacenter, but Cloud operations in general.

On top of that, Amy, along with her director at UCMC Jason Cherry, were attending in order to present their results with VMTurbo as well as the new wave of Automation Amy is leading over there.

About the day

What stood out for me most, was just how informal and social the event was. I thought this was a nice touch compared to other events. It was also extremely customer focused, something I think many other events lose.

The kickoff was great as we learned from CEO Benjamin Nye the direction VMTurbo continues to focus in.

IMG_4670

VMTurbo truly GET that Cloud is real and they are working quickly to adapt to the increased momentum of Public Cloud adoption. VMTurbo treats everything as a commodity and works to create a level of abstraction and allow automated intelligence to determine workload placement.

Given how many vendors have done a poor job at trying to offer automated decisions based on Public Cloud models, I’m excited to see another company with a proven track record in this space on-premises, fully embracing this gap in the market.

The UCMC Presentation

IMG_4678

One of the main reasons I attended was to support my fellow ninja blogger as well as the UCMC team who I’ve had the pleasure of working with.

What strikes me most in all my talks with Amy and Jason is the results they’ve been able to achieve. $600,000 in cost avoidance due to properly utilizing their VMware environments. They saw instant benefits by adding VMTurbo to their infrastructure management tools and were able to get higher density on their clusters as a direct results of the VMTurbo software.

Amy went on to talk about the work around the UCMC Cloud they have been building, primarily on-premises now, with future expansion to Public.

IMG_4680

UCMC will be using a mixture of ServiceNow, vRealize Orchestrator and Puppet to easily automate their deployments and take UCMC IT into the 21st century as they put it. This is similar to the solution we both worked on during our time at a previous employer, and the methodology is clean and simple to use.

IMG_4681

In addition, Amy has been developing a number of vRO Workflows around the VMTurbo plugin specifically for workload placement and chargeback. This placement workflow is called before the VM clone workflow is initiated in order to determine the optimal host location of it. Handy way to integrate the placement decision directly into the workflow instead of having to go to VMTurbo first, get the answer, and then manually key it in as an input. Awesome stuff.

Stay tuned as Amy plans to release all the workflows on our blog when finished.

After the community sessions, the panel sat back down and had a very lively Q&A session with lots of users in the community asking great questions.

IMG_4683

Before I left, the VMTurbo marketing team played one of the most impressive videos I’ve honestly ever seen. I’m trying to get a youtube link of it to post here as it is pure awesome. Think VMs meet insane action movie with awesome effects.

ClBH0_9WAAE9lOf

With that said, I can’t wait to do more with VMTurbo again myself, and look forward to catching up with the team again at VMworld.

 

PowerCLI: Checking for and removing Virtual Machine Memory Limits

Here is a quick one liner I found to check for any VMs which had memory limits set on them:

Get-VM | Get-VMResourceConfiguration | where {$_.MemlimitMB -ne -1}

If you want to target a specific cluster, just add Get-Cluster “clustername” to the beginning:

Get-Cluster “Clustername” | Get-VM | Get-VMResourceConfiguration | where {$_.MemlimitMB -ne -1}

Now if you want to get rid of the memory limits, add the following:

Set-VMResourceConfiguration -MemlimitMB $null

Final script for all VMs to find and remove limits:

Get-VM | Get-VMResourceConfiguration | where {$_.MemlimitMB -ne -1} | Set-VMResourceConfiguration -MemlimitMB $null

Next step…setting this as a scheduled workflow in Orchestrator to run every night/week and send a report out of any limits discovered.

*Note: This is a repost due to move from Systemsgame.com to 2ninjas1blog.com*

PowerCLI: List of VMWare Hosts, Clusters, Datacenters

Just a quick one liner I used to gather a list of VMWare Hosts including their cluster and datacenter.

get-vmhost | Select Name, @{N=”Cluster”;E={Get-Cluster -VMHost $_}},@{N=”Datacenter”;E={Get-Datacenter -VMHost $_}} | Export-csv c:\temp\inventory.csv

 

*Note: This is a repost due to move from Systemsgame.com to 2ninjas1blog.com*

IaaS Fundamentals: Creating a fresh Windows Server 2012 Template – Part 2

With our base VMware vSphere VM shell ready, it’s time to continue installing the Windows OS.

rwc-template-winstart2

Just before we dive in, it is worth noting that depending on how you are remotely connected into the desktop, you may have issues controlling your mouse. In my case I was going via a View Desktop and then into the VRM console. I decided to just use the Tab and Spacebar key instead to make my selections. This will get much easier later on when VMware Tools is installed in the VM.

  • Select Install now, accept the defaults for language etc. until you get to the type of OS you wish to deploy.
  • I choose the Datacenter Edition with GUI here. Note: You can always remove the GUI and go back to ServerCore if needed. I know in my environment our Windows team still generally uses the GUI
  • Click Next once chosen

rwc-template-instancetypes

 

  • Accept the license terms and click Next
  • Change the installation type to Custom: Install Windows Only (advanced) and click Next.

rwc-newtemplate-installtype

 

  • Next you will be prompted for your drive layout. It should look like the screenshot below unless you chose a different drive configuration.

rwc-template-driveallocation

  • Leave Drive 0 selected and click Next

Sit back relax and enjoy the show!

rwc-template-installingzzz

 

Enjoy some tea while you wait…

tea

  • Once finished you will need to enter your Administrator Password for your Windows Template.

Coming soon – Part 3 – Configuring and tuning your OS

Rubrik Announces r528 Cloud Appliance and Sexy New Features

Rubrik announced the r528 cloud appliance today.  Yes, Rubrik just got sexier. Not only has Rubrik grown exponentially as a company ,they are on their  3rd update and are now quite the global force with 90+ signed Channel Partners and 4PB+ Protected in the Field

r528

The r528 offers encryption at rest and in flight from VMware.  Because the appliance is using hardware encryption, there is no compromise on speed or performance.  The self encrypting drives (SED), use  AES 256 circuitry. All data written to disk is encrypted automatically and data read is decrypted automatically.  Eliminating or overwriting the security key would perform an instantaneous wipe.  If a drive were to be taken out, it would be deemed worthless without the key

r528-2

Boring stuff you should know about: NIST

rubrik-fips

 

This offering is FIPS 140-2 Security validated. What does that mean? It sounds important.  The drives and Rubrik Cryptographic library are FIPS 140-2 certified.  Where most backup appliances are Level 1, Level 2 brings about the ability to detect physical tampering.  If you want to nerd out and read up on FIPS 140 here. From there, you can read that FIPS 140-2 Level 1 provides the lowest level of security. Basic security requirements are specified for a cryptographic module (e.g., at least one Approved algorithm or Approved security function shall be used). No specific physical security mechanisms are required in a Security Level 1 cryptographic module beyond the basic requirement for production-grade components. FIPS 140-2 Level 2 improves upon the physical security mechanisms of a Security Level 1 cryptographic module by requiring features that show evidence of tampering, including tamper-evident coatings or seals that must be broken to attain physical access to the plaintext cryptographic keys and critical security parameters (CSPs) within the module, or pick-resistant locks on covers or doors to protect against unauthorized physical access.

For key management, Rubrik supports external key standards using KMIP 1.0 or they also provide a Trusted Platform Module (TPM) so there would be no need for KMS .  Giving the customer options if they don’t have KMS setup in their environment.

But wait there’s more!

Rubrik Converged Data Management 2.2

Enhancing the auto protect and SLA inheritance Rubrik already offers

  • Dynamic Assignment – Set policy on a vCenter, Data Center, Cluster, Folder, Host, and more.
  • Inheritance Options – Any new object or workload created will automatically pick up parent SLA assignment.
  • Do Not Protect – Block SLA policy from being inherited with explicit denial to prevent data protection at any desired level.

Throttle detection!

Most people don’t want backups to affect workloads.  The software can look for latency to make sure it’s not causing performance issues. If storage latency is rising, it is smart enough to halt additional tasks on the fly.  Backups don’t continue to pile up on your environment like a WWE royal rumble.

Battle_crop_north

Scalability

  • In testing, Rubrik has scaled out to a 20u, 10 brik, 40 node cluster.  That’s insane
  • Protect 10,000 VMS using vSPhere 6.0
  • Instant recovery:  Quicker spin up of clone workloads (thanks to being able to get 20,000 I/O per brik) and faster storage vMotion to your production environment.

Cluster Policy Enhancements

  • Global pause gives you the ability to use a maintenance window to perofrm work on the cluster
  • Recurring First Full Snapshot Window gives you control to say when a full backup should be performed within an SLA
  • New Retention Periods bring increased flexibility for SLA policies to meet different customer requirements
  • Blackout Windows define when no operational taks should be executed by the cluster
NAT Support
For customers that don’t want to use site-to-site tunneling , there is now NAT support for public bi-directional replications.

 

User Experience and Management
You might as well enjoy managing your backups …userexp
Last but not least
Backup physical alongside your virtual environment.  This includes SQL and Linux. You have all the capabilities for physical recovery that you are used to having for your VMs

Automation Fun

automate_all_the_things

An oldie but goodie, you can tell from this blog, we’re all about automating all the things possible.

There are several options to satisfy the automation ninja within you:

GitHub PowerShell-Module Repository
PowerShell Gallery (NuGet)
Continuous Integration with AppVeyor

And a personal favorite:
vRealize Orchestrator Packages

Rubrik maintains it’s mantra don’t backup go forward with it’s continuous improvements to the backup experience.

Server Name Generator – Final

Now that we have our partial name creator and our workflow to check for the next available name, we can bring it all together for a complete server name workflow.

Complete Server Name Workflow

General Attributes:

  • partialName: Type = String

 Inputs:

  • appType: Type = String
  • location: Type = String
  • network: Type = String
  • envLevel: Type = String
  • OS: Type = String
  • domainSuffix = string

Outputs:

  • vmName: Type = String
  • fqdnOut: Type = String

The Workflow:

completeWorkflow

As explained in Server Name Generator – Part 1, we create the partial name based on datacenter location, network, operating system and environment.  We now append that with a number and check if the name exists in DNS using the worfklow from Server Name Generator – Part 2.

Now the output you will see is something similar to this in the logs:

[2016-04-18 15:01:08.930] [I] Entering Generate Partial Name Worfklow
[2016-04-18 15:01:08.935] [I] The datacenter shortname is AKL
[2016-04-18 15:01:08.948] [I] The network shortname is P1
[2016-04-18 15:01:08.981] [I] The OS shortname is W
[2016-04-18 15:01:08.994] [I] The environment shortname is 1
[2016-04-18 15:01:09.013] [I] The shortname of the application is APP
[2016-04-18 15:01:09.028] [I] The partial name of the VM is AKLP1W1APP
[2016-04-18 15:01:09.030] [I] Leaving Generate Partial Name Workflow
[2016-04-18 15:01:09.072] [I] Your partial name is: AKLP1W1APP
[2016-04-18 15:01:09.073] [I] Starting loop --- 
[2016-04-18 15:01:09.075] [I] Server Name: AKLP1W1APP01 found in master list - Incrementing with next number and starting over
[2016-04-18 15:01:09.078] [I] Server name: AKLP1W1APP02 not found in master list. Recording new name and continuing.
[2016-04-18 15:01:09.079] [I] Checking host for existing DNS record: AKLP1W1APP02.tritech.local
[2016-04-18 15:01:09.086] [I] Check complete for FQDN: AKLP1W1APP02.tritech.local Found IP address of: null
[2016-04-18 15:01:09.588] [I] 
[2016-04-18 15:01:09.591] [I] The vmName to pass as output is: AKLP1W1APP02
[2016-04-18 15:01:09.592] [I] The FQDN to use is: AKLP1W1APP02.tritech.local

AKLP1W1APP01 already existed so our loop went to the next number 02 and found that it didn’t exist.

This concludes the Server Naming series. You can download the complete workflow here.  If you have any questions, or found this content useful, let us know with a comment.

 

IaaS Fundamentals: Creating a fresh Windows Server 2012 Template – Part 1

Now that we’ve set our approach for template creation and management, it’s time to create our on-premises vSphere template for Windows Server 2012. The example below is based on VMware vSphere as the hypervisor of choice.

Things you will need

  1. Microsoft Windows Server 2012 ISO (Download from Microsoft)
  2. License Key
  3. vCenter Access

Step 1: Create your ISO

It really helps to first upload your ISO to one of your datastores. Many people prefer an NFS store attached to vSphere for this purpose as it it allows more flexibility when you want to connect that ISO to multiple other hosts where your storage array may not be mapped. In our lab examples, we are using a Tintri-T880 VMstore to keep our templates.

Create a Folder and Upload Windows ISO

  • Login to vCenter
  • Browse to your Datastore
  • Create a New Folder (Something like ISOs or Windows ISOs so you can find it easily)

rwc-w2k12-datastoreadd

  • Browse to your new folder
  • Select Upload File to Datastore
  • Browse for your ISO and let it upload. This will take a short while depending on your connection to vSphere

Step 2: Create your VM Shell

  • Switch back to VMs & Templates View
  • Create a new folder to store your templates in

rwc-template-newfolder-1

 

  • With your new folder selected, select Create a New Virtual Machine from the actions menu

rwc-template-newVM-1

  • Select Create a new virtual machine and click next
  • Type in a name for your template. In our example we use “TT_W2K12_Template”. Simple and easy to find. Select the VM Templates folder you created previously and click next.
  • Select your vSphere Cluster and click next
  • Select your Datastore and click next
  • Choose your compatibility level. Our clusters are all at 5.5 or above so we have no issues selecting the default of ESXi 5.5 and later.
  • Customize your VM Hardware
    • Choose your CPU, Memory, and Disk configuration
      • 1 CPU
      • 4096 MB Memory
      • 60GB Disk (Up from the default of 40GB. Many could argue to keep it at 40, but with patches and other functions in Windows Management I’ve found 60 to be a safer amount. Plus, since I’m thin provisioned on the storage, it adds little additional cost to me)
    • Change the network adapter to VMXNET3 (The days of needing to use the E1000 are over and 2012 supports the VMXNET3 fine without needing to install VMware tools first)

rwc-newtemplate-customizehw

  • Attach the ISO we created earlier to the CD-ROM

rwc-newtemplate-iso

  • Select the VM Options Tab and change the boot options so that the VM boots to the BIOS first.

rwc-template-bios

 

  • Power On the VM and it should be automatically in the BIOS
  • Go to Advanced > I/O Device Configuration and disable the Floppy Drive, Serial Ports, and Parallel Port

rwc-newtemplate-disableio

 

  • Press F10 to Save and Exit
  • Edit Settings on your VM and connect the CD Rom Drive.

rwc-template-cdconnect

  • Restart the VM to begin installing the OS

rwc-template-winstart1

This concludes the template prep, in part 2 we will continue installing the OS.

 

Server Name Generator – Part 2

Now that we have our partial name being created from Server Name Generator – Part 1, we can have a workflow that appends with a number and check to see if that name already exists within the environment.  The server names will be checked via DNS and via a text file named names.txt saved on the vRealize Orchestrator appliance.

First some pre-work needs to be done.  We need to be able to add host names as they are created to the “names.txt” file

1. Login as root onto the appliance
2. Locate and view the “js-io-rights.conf” file by running “: less /etc/vco/app-server/js-io-rights.conf.  The purpose of this file is explained here on VMware’s documentation site.

addline

You will see “+rwx /var/run/vco/” as shown above.  This means vRO can read, write and execute from this location.  This is where will place our text file to write host names to.

3.  CD to var/run/vco and do a “touch names.txt” to create the file in that directory.

4.  As you can see we need to change the rights so run: chmod 666 names.txt to give read/write access to vRO.

names

Server Name Generator Workflow

General Attributes:

  • None

 Inputs:

  • partialname: Type = String
  • domainSuffix: Type = String

Outputs:

  • vmName: Type = String
  • fqdnout: Type = String

The Concept:

CODE

The Code:

// Read temp server name list - prepare to invoke file writing capabilities if needed
var fr = new FileReader("/var/run/vco/names.txt");
var fw = new FileWriter("/var/run/vco/names.txt");

fr.open();
var content = fr.readAll();
fr.close();

// Initialize variables
var number = 1 ;
var temphost = 0 ; // If we don't set temphost to zero, we break out of for loop immediately


// Function to pad zeros on number that we increment while number is less then 10
function padzero(number) {
 return (number < 10 ? '0' : '') + number
}

// increment through hostnames until we find a hostname that does not exist

System.log("Your partial name is: " + partialName) ;
System.log("Starting loop --- ") ;

for (number=1; temphost != null; number++)
{
 var padded_number=padzero(number) ;
 var vmName = partialName + padded_number;
 
 if(content.search(vmName) <0)
 {
 System.log("Server name: " + vmName + " not found in master list. Recording new name and continuing.") ;
 fw.open() ;
 fw.writeLine(" " + vmName) ;
 fw.close() ;
 
 var fqdn = vmName + "." + domainSuffix ;
 
 System.log("Checking host for existing DNS record: " + fqdn) ;
 var temphost = System.resolveHostName(fqdn) ;
 
 
 
 System.log("Check complete for FQDN: " + fqdn + " Found IP address of: " + temphost) ;
 System.sleep(500) ;
 }
 else
 {
 System.log("Server Name: " + vmName + " found in master list - Incrementing with next number and starting over") ;
 temphost != null ;
 }
}

// Log for debugging purposes
//Output FQDN
fqdnOut = fqdn;

System.log("") ;
System.log("The vmName to pass as output is: " + vmName);
System.log("The FQDN to use is: " + fqdnOut) ;

Download the workflow here

In the next post, we will put the two workflows together to pass the partial name to the server name generator for a completely automated name.

Automating Tintri with vRA and vRO – Workflow Updates and Webinar

Screen Shot 2016-04-05 at 3.02.54 PM

 

A while back I wrote some workflows with Adam Cavaliere on Automating Tintri with VMware vRealize Orchestrator. These were posted on my old blog and can still be found here. So you know where to find things going forward, all of this content is in the process of being moved here to 2ninjas1blog.com via this new blog series.

We showed demos of those workflows to hundreds of our customers at Ahead and Tintri and received tons of great feedback. There were things we didn’t get around to the first time and finally we have been able to do some big updates which we think you will find extremely useful.

Over the coming weeks, we will be releasing blog posts showing how to use the new workflows. These posts will help you configure the workflows and get up and running with them quickly. Furthermore, Adam and I are hosting a joint Ahead/Tintri Webinar on April 7th where you can see all of these workflows in action.

tintriwebinar

So what are the updates:

  • Updated Tintri REST Login Workflow and Actions
  • Updated Snapshot workflow with additional error handling
  • Replication Workflow – Setting up a replication policy and triggering replication on demand
  • Sync VM Updates – Updates and improvements to these workflows
  • Sync VM and Data Sanitization Example – Using vRO to extend Sync VM so it sanitizes PII data
  • Integrating with ServiceNow
  • vRealize Automation Examples of Day 2 Operations – Long overdue and something I showed a few people some time ago, but this makes it real and ties into the Hybrid Cloud series Amy and I are writing.
  • Multiple other small code updates to improve reliability and support Tintri Global center

We hope you will agree there is lots of great content and useful workflows here. Over 500 people downloaded the workflows before and we hope you can continue to give us feedback so we can continue to improve them.

Where can I get them?

For right now, the packages are being uploaded to GitHub.

https://github.com/nickcolyer/tintri-vro-workflows-v2

Please comment here or on the specific blog posts that come out with any feedback on the specific modules.

Stay tuned for more info coming soon on here and at Tintri and Ahead.

Real World Cloud: Template Creation and Management

To bake in, or not to bake in, that is the question you must ask.

rwc-keepitsimple

Keeping it simple…

Without a doubt, template creation and template sprawl are items people worry about when getting ready for their cloud environment. Maybe you have a Linux team that use PXE plus kickstart in the past and you’re having to convince them to move to templates, or maybe you had a bunch of templates with everything baked inside. When it comes to cloud, I’m personally a big believer in keeping your template as bare as possible, providing it doesn’t add a significant overhead to the build time.

This means stripping the template down to the base OS and layering everything on top.

The key also to remember here is that you want to try to be able to use a similar approach to your template management in your Public Cloud, as well as, your On-Premises Datacenter. This way you aren’t managing 2 completely separate processes and creating more complexity.

How do I configure things like agents, settings etc. after the template is cloned?

The rise of configuration management tools in the past few years has been key. Puppet and Chef in particular have become extremely widespread in the past few years. On top of that Microsoft is constantly evolving PowerShell DSC, and we have other newcomers like Saltstack and Ansible that are quickly gaining traction.

Essentially, everything you install and configure on top of the VM deployed, will be via Configuration Management. This allows us to completely abstract the configuration from the virtual machine. Think of it like a container (not to be confused with docker), that just hosts a specific configuration. The more you can modularize the pieces that sit on top of the OS, the easier it is to manage. This does however come at a cost of speed, because each application and potentially a number of reboots will need to be configured post VM provisioning.

rwc-vmtemplatelayers

How are Configuration Management Tools Changing?

I’m fortunate enough to work with some great Puppet engineers at Ahead who work on these pieces daily. Rather than retype what one of my colleagues at Ahead has already written, I suggest you check out this blog article on the Ahead Blog. http://blog.thinkahead.com/the-present-and-future-of-configuration-management

Creating the Template:

There are a number of ways to do this, and i’m sure the are more than the ones I list here:

  1. vSphere: Clone from an Existing VM and prep it.
  2. vSphere: Create a fresh template from an ISO.
  3. Amazon: Create a Windows AMI from a running instance. (Use a prebuilt Amazon AMI basically and customize it)
  4. Amazon: Import your vSphere VM and convert.

In the next post, we will be drilling down in detail on creating a fresh Windows vSphere Template for use in your Cloud. As we progress to Public Cloud, we will come back to the Amazon AMI options.