Provisioning NetApp Storage w/ DataONTAP Powershell Module

The following script can be used to provision new Volumes and LUN’s on a NetApp storage system. Please view the notes at the end of this article for additional information.

I’d like to note a lesson that I learned when working with NetApp storage. When you are using NetApp storage systems in conjunction with NetApp SnapDrive (the server side storage agent and provisioning software) creating and attaching raw LUN’s to a Windows based server operating system SnapDrive may not automatically pick up the new LUN’s. Based on some research I did when I discovered this rather annoying ‘issue’ SnapDrive is looking for fully formatted disks ready to be attached. Because these are brand new raw LUN’s which have not been initialized and formatted by a host operating system SnapDrive times out operations to add these LUN’s to its configuration. This is an equally annoying workaround listed here.

With the above being true this script should ultimately be written to include all of the necessary steps to not only create and attach the LUN’s but to reach out to the servers to which the LUN’s are being attached, discover the newly attached LUN’s and complete the initialization and disk volume creation process. I may do this in the future…time allowing.

The table below contains the fields I’ve included in the CSV config file I use with the script. Not all fields are directly called or referenced by the script but are used for documentation and planning purposes.

FIELD NAME
DESCRIPTION
SERVERThis field is not referenced in the script but is useful for tracking your VOL and LUN layout and assignment. It is similar to the "VMNAME" field and will often contain the same data. This field could be useful in cases where you are attaching disk to a VM Host in which case you may then use the "VMNAME" field to track which Virtual Machine the disk will be used for or on.
VMNAMEThis field is not referenced in the script. Reference the "SERVER" field description for additional information.
LUNDESCThis field is not referenced in the script. Contains a description or "name" for what the LUN is intended for. I personally use this to create the VOLNAME, QTREENAME, and LUNNAME fields. This process is not automated at this time and must be done manually in Excel or a text editor.
CTRLNAMEThis field is not referenced in the script. For documentation purposes this field contains the name of the NetApp head unit on which you will be provisioning storage. This field could be used if you have proper DNS entries for your NetApp heads.
CTRLIPThis field is used to determine which NetApp head you will be connecting and acting against. You could use an IP address or an DNS name if that is available in your environment.
AGGRThis field contains the Aggregate name on which you will be provisioning the VOL and LUN.
VOLNAMEThis field contains the name of the Volume to be created. The volume name can contain letters, numbers, and the underscore character (_),but the first character must be a letter or an underscore. If you dont know already.. do NOT use dashes/hyphens as they are not allowed.
VOLSIZEThis field contains the size for the Volume being provisioned. A valid format for this field is a numerical size followed by a measurement unit. Example: [k|m|g|t] where "k" means kilobytes, "m" means megabytes, "g" means gigabytes, and "t" means terabytes. If no units are specified, the number is interpreted as the number of bytes.
VOLMAXThis field contains the volume's maximum allowed size as part of the Volume AutoGrow configuration. A valid format for this field is a numerical size followed by a measurement unit. Example: [k|m|g|t] where "k" means kilobytes, "m" means megabytes, "g" means gigabytes, and "t" means terabytes. If no units are specified, the number is interpreted as the number of bytes.
VOLGROWThis field contains the volume's AutoGrow increment size. If the system performs an AutoGrow of the volume it will use this setting to determine the size to which the volume should be increased assuming that the increase falls within the volume maximum allowed size. A valid format for this field is a numerical size followed by a measurement unit. Example: [k|m|g|t] where "k" means kilobytes, "m" means megabytes, "g" means gigabytes, and "t" means terabytes. If no units are specified, the number is interpreted as the number of bytes.
QTREENAMEThis field contains the path of the qtree to create, in the form /vol//. Generally I create the qtree with a name that includes the Volume name with a suffix of "q_" attached.
LUNNAMEThis field contains the path of the LUN you wish to create. The format is /vol///. This is generally the previous Qtree field with "/.lun" attached.
LUNSIZEThis field contains the size for the LUN being provisioned. A valid format for this field is a numerical size followed by a measurement unit. Example: [k|m|g|t] where "k" means kilobytes, "m" means megabytes, "g" means gigabytes, and "t" means terabytes. If no units are specified, the number is interpreted as the number of bytes.
LUNTYPEThis field contains the the OS type for the LUN. Currently, supported ostypes are "solaris", "windows", "hpux", "aix", "linux", "netware", "vmware", "windows_gpt", "windows_2008" (the lun will be used to store Windows data on Windows 2008 systems), "openvms", "xen", "hyper_v" (the lun will be used to store Hyper-V data), "solaris_efi", "image" (the default ostype, indicating that no assumptions can be made about the contents in the lun). *NOTE* It is strongly recommended to avoid using the "default" type because it could result in misconfigured LUNs. For example, a lun with ostype "default" could suffer major performance penalties when a Windows host is trying to access it. Please refer to your configuration/design requirements and appropriate NetApp documentation.
IGROUPThis field contains the name of the Initiator Group to be created.
igProtocolThis field contains the type of Initiator Group to be created. Currently only "iscsi" and "fcp" are supported.
igTypeThis field contains the OS type of the Initators which will be added to the Initator Group. The type applies to all initiators within the group and governs the finer details of SCSI protocol interaction with these initiators. Valid arguments are "default", "solaris", "windows", "hpux", "aix", "linux", "netware" and "vmware". *NOTE* It is strongly recommended to specify an OS type that is not "default". Some host OSes require this type field be set correctly in order to function properly.
iGroupInitatorThis field contains the Initiator you wish to have added to the previously created Initiator Group. For iSCSI use this would be the iqn which is configured on the target server. Example: iqn.1992-05.com.microsoft:aservernamedsue.company.com

Requirements: Data ONTAP Powershell Toolkit

#Start Logging
	$logSerial = Get-Random -Minimum 10000 -Maximum 99999
	$date = Get-Date -format "yyyyMMdd"

	#Set Log Path - do not change filename
	$LogPath = "c:\netapp-ps\StorageProvisioning-$logSerial-$date.txt"

	Start-Transcript -path $LogPath

#Load DataONTAP PoSH Module

	Import-Module DataONTAP

#Load Config CSV
	$luns = Import-Csv c:\netapp-ps\config.csv

# Prepare Auth
	$password = ConvertTo-SecureString "password" -AsPlainText –Force
	$cred = New-Object -TypeName System.Management.Automation.PSCredential -ArgumentList "root",$password

#Create iGroups and iGroupIniators	
	ForEach ($lun in $luns){

		Connect-NaController -Name $lun.ctrlip -Credential $cred -HTTPS

		New-NaIgroup $lun.igroup $lun.igProtocol $lun.igType
		Add-NaIgroupInitiator $lun.igroup $lun.iGroupInitator
	}

#Create VOLs, Set VOL Options, Set Snapshot options, Create Qtree, Create LUN
	ForEach ($lun in $luns){

		Connect-NaController -Name $lun.ctrlip -Credential $cred -HTTPS

		New-NaVol -Name $lun.volname -Aggregate $lun.aggr -Size $lun.volsize -SpaceReserve "none"
		Set-NaVolAutosize -Name $lun.volname -EnableGrow -MaximumSize $lun.volmax -IncrementSize $lun.volgrow 
		Set-NaSnapshotReserve $lun.volname 0		
		Set-NaSnapshotAutodelete $lun.volname state on
		Set-NaVolOption $lun.volname fractional_reserve 0		
		Set-NaVolOption $lun.volname convert_ucode on
		New-NaQtree $lun.qtreename
		New-NaLun -Path $lun.lunname -Type $lun.lunType -size $lun.lunsize -Unreserved
	}

#Map Luns to iGroup
	ForEach ($lun in $luns){

		Connect-NaController -Name $lun.ctrlip -Credential $cred -HTTPS

		Add-NaLunMap $lun.lunname $lun.igroup
	}

#End logging

Stop-Transcript
Write-Host "A log of the activities performed is located at:" $LogPath

General Notes:

For any fields where it is applicable please be sure to reference any pre-existing naming conventions used in your environment.

This script was developed for use against standalone NetApp systems and it’s use against clustered NetApp filers is currently untested. The NetApp powershell module contains many Clustered system specific Cmdlets which you may want to review if that is the type of configuration you are using. CmdLets named <verb>-NA… are for non-clustered systems and those beginning with <verb>-NC… for clustered systems.

Connections to the NetApp heads/Controllers use the HTTPS protocol

New Volumes are configured with:

  • Volume AutoGrow – Enabled w/ MaxSize and IncrementSize set in the CSV
  • Space Reserve set to ‘None’
  • Snapshot Reserve set to ’0′
  • Snapshot AutoDelete state of ‘on’
  • Volume Option – Fractional Reserve of ’0′
  • Volume Option – Convert ucode enabled

New LUN’s are created with space unreserved or “Thin Provision” enabled.

The script assumes the use of qTree’s. These can be used for several different things. We happened to need them on the project I was working on that resulted in me creating this script. If you do not need qTree’s you can easily edit this script to remove that portion. Reference your configuration/design requirements and appropriate NetApp documentation.

You are currently required to include the account name and password you wish to authenticate with. This could easily be replaced by modifying the script to include Powershell’s Get-Credential cmdlet. This would result in the script prompting you for credential information which is generally a better security practice as opposed to keeping plain text credential information inside the script. Keep this in mind and do not store this script with the complete credential information for your environment.

I chose not to implement this at this time because I don’t currently run the script in its entirety very often. My common usage is to copy and paste sections into a Powershell console window based on what I need at that time. For instance, after the first run I don’t often need to create iGroup’s and iGroupInitiators again. Future versions, if I create one, may include the option to run the script as a single .ps1 and include parameters to include or bypass certain sections such as iGroup creation.

Script was modified using the recommendation from the comments below to move the credential creation outside of each For loop. Thanks to Jeffrey Snover who is, quite literally, the father of Powershell.

C# – A Small Start

I wrote my first C# program last night. It says hi.

namespace HelloWorld
{
    class Program
    {
        static void Main(string[] args)
        {
            Console.WriteLine("Hello World!");
            Console.ReadLine();
        }
    }
}

C#, or C sharp, is defined by Wikipedia as “a multi-paradigm programming language encompassing strong typing, imperative, declarative, functional, procedural, generic, object-oriented (class-based), and component-oriented programming disciplines.

C# was developed by Microsoft as part of its .net Initiative. You can read more here.

I am following the C# Fundamentals: Development for Absolute Beginners on Microsoft Virtual Academy. MVA is a free learning portal which can be used by anyone…that means you.

CFPB All Up In Your Data: Part 2

Earlier this week I posted an article linking to a story about the Consumer Financial Protection Bureau (CFPB) and their expanded monitoring of bankruptcy data and transactions. In a new article the Washington Examiner provides additional details on the CPFB’s massive credit/debit card transaction data mining efforts.

Consumer Financial Protection Bureau officials are seeking to monitor four out of every five U.S. consumer credit card transactions this year — up to 42 billion transactions – through a controversial data-mining program.

A CFPB strategic planning document for fiscal years 2013-17 describes the “markets monitoring” program through which officials aim to monitor 80 percent of all credit card transactions in 2013.

I’m sure that on several occasions you’ve had a laugh with someone joking about how the government just monitored your last purchase (usually when you are buying something out of the ordinary). Well I suppose most of us kind of new in the back of our heads that it probably wasn’t just a joke. Given recent accountings of massive privacy intrusions by the NSA, FBI, and pretty much every agency from the alphabet soup list no ones laughing any more. The article continued.

At a Wednesday hearing before the House Financial Services Committee chaired by Rep. Jeb Hensarling, R-Texas, CFPB Director Richard Cordray defended the data-mining practice and said his agency is monitoring credit card usage at 110 banks, including Morgan Chase, Bank of America, Capital One, Discover and American Express.

That short list of banks right there covers most Americans. I’m not sure what to think about Visa and MasterCard not being on that list. Nevertheless, one would hope that Representative Hensarling’s constituents see fit to terminate his employment.

Source: WashingtonExaminer.com

An update to: Consumer Financial Protection Bureau all up in your data

[CRITICAL] VMware: Possible data corruption after a Windows 2012 virtual machine network transfer

The kind folks over at Nimble Storage were nice enough to drop me a note this evening regarding an issue regarding VMware e1000e vNIC and iSCSI attached storage on Microsoft Windows Server 2012. The field alert states that:

VMware has confirmed a severe issue that may cause data inconsistency on iSCSI storage systems under the following conditions:
NOTE: This issue is not specific to Nimble Storage

1. Guest OS: Windows 2012 (Windows 8 may also be suspect)
2. Guest NIC emulation type: e1000e
3. TCP segmentation offload (TSO) is enabled
4. ESX 5.0/5.1

VMware says this about the E1000e vNIC as:

E1000e: This feature emulates a newer model of Intel Gigabit NIC (number 82574) in the virtual hardware. This is known as the “e1000e” vNIC. e1000e is available only on hardware version 8 (and newer) virtual machines in vSphere 5. It is the default vNIC for Windows 8 and newer (Windows) guest operating systems.

The fact that this is most likely the default vNIC configured on many newer VM‘s makes this issue all that more important and worth your review. Nimble Storage listed the following preventative actions and next steps:

PREVENTATIVE ACTION:

1. We recommend you immediately stop using e1000e virtual emulation in all cases and use one of the other virtual NICs provided in the guest Virtual Machine.
e.g. VMXNET 3 or e1000 (known to NOT have this issue)

NEXT STEPS:

1. There is no change necessary to Nimble OS. This issue is not specific to Nimble Storage; this issue has been reproduced on other manufacturers iSCSI Storage Arrays.
2. If you are using the configuration shown above, please take necessary steps to confirm the integrity of all data on the iSCSI volumes attached in this manner.

It is VERY important to note that this has been confirmed against multiple storage arrays so this is, as stated, NOT simply a Nimble Storage issue. This is a VMware issue and they have addressed it in the VMware KB2058692 where they state the symptoms as:

On a Windows 2012 virtual machine using the default e1000e network adapter and running on an ESXi 5.0/5.1 host, you experience these symptoms:

Data corruption may occur when copying data over the network.
Data corruption may occur after a network file copy event.

A Satisfying Explanation of Gen Y Unhappiness

I recently stumbled upon this gem of an article while scrolling Facebook. It’s from WaitButWhy.com and it goes into great detail describing what the author refers to as “Gen Y Protagonists & Special Yuppies” or GYPSY’s for short. The article gives you a high level overview of why a fictional GYPSY named Lucy is unhappy.

Lucy is unhappy, basically, because of improperly managed expectations. As with so many things in this world LIFE itself requires properly managed expectations in order to be perceived as a “happy” thing. The author offers this graphic in reference to Lucy’s unrealistic career expectations:

looking up

I agree very much with what the article states. Many in my generation seem to lack the ability to keep themselves centered around reality. Many have been ushered through their early life while receiving gold starts and trophies for losing. The mantra of “you can be anything you want” while partially true is often understood as “you are owed anything you want.”

Previous generations were required to work much much harder to achieve goals and overall success. Being an exceptional person was actually that…being exceptional. In this day and age children are raised to believe that not only are they exceptional simply because they are alive but that they should expect exceptional things to happen because they deserve it. They deserve a big house, expensive car, lavish vacations, and a gift wrapped super-career with relatively little effort applied.

As seen in the article this leaves many staring up at a rainbow barfing unicorn, while being unhappy with their position in life, and frustrated with those around them who have done better simply because GYPSY’s have been raised to think that everyone gets the gold star in the end.

The author ends with these three bits of advice:

  • Stay wildly ambitious
  • Stop thinking that you’re special
  • Ignore everyone else

Society in general could use some advice. Stop giving every kid a gold star just for showing up. Sometimes you win. Sometimes you lose. People keep score during sporting events. You will probably fail more often than you succeed. Greater success often comes from painful failures. No one owes you anything.

Source: WaitButWhy.com

Link

Bruce Schneier on Our Newfound Fear of Risk

When you implement measures to mitigate the effects of the random risks of the world, you’re safer as a result. When you implement measures to reduce the risks from your fellow human beings, the human beings adapt and you get less risk reduction than you’d expect — and you also get more side effects, because we all adapt.