Volumes are the containers of data in a NetApp storage system. They are “stored” on aggregates, accessed via Storage Virtual Machines, and are the point-of-application for many of the features of Data ONTAP. Let’s look at what we can do with volumes leveraging the PowerShell Toolkit:
- Creating, Deleting, and Re-sizing Volumes
- Volume Features
- Thin Provisioning
- Deduplication
- Compression
- AutoGrow / AutoShrink
- Fractional Reserve
- Quality of Service
- Volume Options
- Snapshots
- FlexClones
- Volume Move
IMPORTANT! It is VERY IMPORTANT that you are mindful of the SVM context for most of these commands. If you have two volumes with the same name in different SVMs, and you don’t specify the SVM, the action will affect both. This is no different than the CLI, where you must provide the SVM to be modified.
I am not specifying the SVM with most of the commands against volumes for the sake of brevity, however please, please (please!) keep this in mind as you perform actions against your volumes.
For any action which affects volumes, there will be a parameter named “VserverContext”. This is the parameter that you will want to specify:
1 |
Get-NcVol -VserverContext "SomeDataSVM" -Name $volumeName |
This command can now, safely, be pipelined into others which modify properties.
Creating, Deleting, and Resizing Volumes
Creating a volume is an easy enough operation and works very similarly to the CLI.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 |
# create a new volume for a SVM $splat = @{ # a unique name for the volume 'Name' = $volumeName; # the aggregate to create the volume on 'Aggregate' = $aggrname; # the junction point for the volume. I find using the # volume name to be the easiest 'JunctionPath' = "/$volumeName"; # unix or ntfs, depending on the type of system # accessing the data. don't use mixed unless you are # in the process of transitioning 'SecurityStyle' = 'unix'; # for VMware volumes I do not enable automatic snaps, # instead relying on VSC's Backup and Recovery plugin 'SnapshotPolicy' = 'none'; # none = thin provision the volume # volume = thick provision the volume (default) 'SpaceReserve' = 'none'; # the amount of space reserved for snapshots 'SnapshotReserve' = '0'; # the size of the volume. can be appended with "g" # for GB, "t" for TB, etc. 'Size' = "$($volSize)g"; } Get-NcVserver $svmName | New-NcVol @splat |
We can also modify volume properties using the NPTK:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
# Online, offline, or restrict a volume Get-NcVol $volumeName | Set-NcVol -Online Get-NcVol $volumeName | Set-NcVol -Offline Get-NcVol $volumeName | Set-NcVol -Restricted # modify the size of a volume to an absolute size. don't forget that this # will not affect LUN sizes, just the volume! Get-NcVol $volumeName | Set-NcVolSize -NewSize 100g # modify the size of a volume using a relative amount Get-NcVol $volumeName | Set-NcVolSize -NewSize +10g # modify the size of a volume using a relative size. this is particularly # helpful when you know you want to keep a certain amount of free space, # e.g. 20%, where you can just add a % of space Get-NcVol $volumeName | Set-NcVolSize -NewSize +20% |
If your volume is a SnapMirror source volume, remember that the destination will increase it’s size as well.
Volume Features
- Thin Provisioning
123456789# enable thin provisioningGet-NcVol $volumeName | Set-NcVolOption -Key guarantee -Value none# disable thin provisioningGet-NcVol $volumeName | Set-NcVolOption -Key guarantee -Value volume# find thick provisioned volumes, enable thin provisioningGet-NcVol | ?{ ($_ | Get-NcVolOption -Hashtable).value.guarantee -ne "none" } |Set-NcVolOption -Key guarantee -Value none - Deduplication
123456789101112131415161718# get deduplication statusGet-NcVol $volumeName | Get-NcSis# show space saved as a result of deduplication# this one is a bit odd, the Get-NcEfficiency cmdlet returns# an object for the "Returns" property, which contains the# information we want(Get-NcEfficiency $volumeName).Returns.Dedupe / 1gb# enable deduplicationGet-NcVol $volumeName | Enable-NcSis | Start-NcSis# set deduplication scheduleGet-NcVol $volumeName | Set-NcSis -Schedule "sun-sat@2"# scan entire volume, this is required if you enable deduplication# on a volume that has preexisting data in itGet-NcVol $volumeName | Start-NcSis -Scan - Compression
12345678910111213141516# show status, this is a long command, but makes# the output much easier to readGet-NcVol $volumeName |Get-NcSis |Select Path,@{N="Compression"; E={ $_.IsCompressionEnabled }}, `@{n="Inline Compression"; E={ $_.IsInlineCompressionEnabled }} |Format-Table -AutoSize# enable post processGet-NcVol $volumeName | Set-NcSis -Compression:$true# enable inlineGet-NcVol $volumeName | Set-NcSis -InlineCompression:$true# disable compressionGet-NcVol $volumeName | Set-NcSis -Compression:$false -InlineCompression:$false - AutoGrow / AutoShrink
123456789101112131415161718192021# get the current min/max sizeGet-NcVol $volumeName | Get-NcVolAutoSize# set the min/max size$splat = @{# the volume to modify"Name" = $volumeName;# grow_shrink will enable both actions"Mode" = 'grow_shrink';# set the minimum and maximum sizes"MinimumSize" = $minSize;"MaximumSize" = $maxSize;}Set-NcVolAutosize @splat# set the size at which the volume will grow/shrinkGet-NcVol $volumeName | Set-NcVolAutosize -GrowThresholdPercent 97 `-ShrinkThresholdPercent 85 - Fractional Reserve
Fractional reserve is the amount of volume space that is reserved for LUN writes when a snapshot is taken. It is only applicable when the volume contains one or more LUNs. In recent versions of Data ONTAP there are only two available values: 0% and 100%.12345# view fractional reserveGet-NcVol $volumeName | Get-NcVolOption | ?{ $_.Name -eq "fractional_reserve" }# set fractional reserve, 0 = off, 100 = onGet-NcVol $volumeName | Set-NcVolOption -Key fractional_reserve -Value 100 - Quality of Service
QoS was added to clustered Data ONTAP 8.2 and is an extremely helpful feature for a couple of reasons. One of those is the obvious ability to limit the amount of IOPS or throughput that a volume can use. The other less obvious use is workload characterization. QoS collects many statistics about the monitored workload and can report information like IO size, R/W mix, and much more.1234567# create a QoS policyGet-NcVserver $svmName | New-NcQosPolicyGroup -Name $qosPolicyName -MaxThroughput INF# modify limits, remember a QoS policy can be either IOPs or bits,# but cannot be bothSet-NcQosPolicyGroup -Name $qosPolicyName -MaxThroughPut "1000IOPS"Set-NcQosPolicyGroup -Name $qosPolicyName -MaxThroughput "1gb/s"Assigning a QoS policy to a volume is not quite so graceful, so let’s create a couple of functions to make it easier:
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148function Set-NcVolQosPolicyGroup {[CmdletBinding(SupportsShouldProcess=$true)]param([parameter(Mandatory=$true,ValueFromPipeline=$true,ValueFromPipelineByPropertyName=$true)][String]$Name,[parameter(Mandatory=$true,ValueFromPipeline=$true,ValueFromPipelineByPropertyName=$true)][String]$PolicyGroup,[parameter(Mandatory=$false,ValueFromPipelineByPropertyName=$true)][Alias("Vserver")][String]$VserverContext = $null)process {# verify the volume$volume = Get-NcVol -Name $Name -VserverContext $VserverContextif (!$volume) {throw "Unable to find volume with name $($Name)"}# verify the QoS Policy Group$policy = Get-NcQosPolicyGroup -Name $PolicyGroupif (!$policy) {throw "Unable to find policy group with name $($PolicyGroup)"}# a query for the update action$query = Get-NcVol -Template# initialize the search for the volume we wantInitialize-NcObjectProperty -Object $query -Name VolumeIdAttributes# specify we want to operate on the provided volume$query.VolumeIdAttributes.Name = $volume.Name# initialize the update template$attributes = Get-NcVol -Template# initialize the QoS attr propertyInitialize-NcObjectProperty -Object $attributes -Name VolumeQosAttributes$attributes.VolumeQosAttributes.PolicyGroupName = $PolicyGroup# update the volumeif ($PSCmdlet.ShouldProcess($volume,"Attach policy group $($policy.PolicyGroup).")) {Update-NcVol -Query $query -Attributes $attributes | Out-Null}$volume | Get-NcVolQosPolicyGroup}}function Remove-NcVolQosPolicyGroup {[CmdletBinding(SupportsShouldProcess=$true)]param([parameter(Mandatory=$true,ValueFromPipelineByPropertyName=$true)][String]$Name,[parameter(Mandatory=$false,ValueFromPipelineByPropertyName=$true)][Alias("Vserver")][String]$VserverContext)process {# verify the volume$volume = Get-NcVol -Name $Name -VserverContext $VserverContextif (!$volume) {throw "Unable to find volume with name $($Name)"}# a query for the update action$query = Get-NcVol -Template# initialize the search for the volume we wantInitialize-NcObjectProperty -Object $query -Name VolumeIdAttributes# specify we want to operate on the provided volume$query.VolumeIdAttributes.Name = $volume.Name# initialize the update template$attributes = Get-NcVol -Template# initialize the QoS attr propertyInitialize-NcObjectProperty -Object $attributes -Name VolumeQosAttributes$attributes.VolumeQosAttributes.PolicyGroupName = "none"# update the volumeif ($PSCmdlet.ShouldProcess($volume,"Remove policy group.")) {Update-NcVol -Query $query -Attributes $attributes | Out-Null}$volume | Get-NcVolQosPolicyGroup}}function Get-NcVolQosPolicyGroup {[CmdletBinding()]param([parameter(Mandatory=$true,ValueFromPipelineByPropertyName=$true)][String]$Name,[parameter(Mandatory=$false,ValueFromPipelineByPropertyName=$true)][Alias("Vserver")][String]$VserverContext)process {Get-NcVol -Name $Name -VserverContext $VserverContext | Select-Object `Name,@{N="Policy Group Name"; E={ $_.VolumeQosAttributes.PolicyGroupName }}}}And now, with our functions, we can show, add, and remove QoS Policy Groups from volumes easily.
12345678# get the policy group for all volumesGet-NcVol | Get-NcVolQosPolicyGroup# remove a policy group for some volumesGet-NcVol $volume1,$volume2 | Remove-NcVolQosPolicyGroup# set the policy group for a volumeGet-NcVol $volumeName | Set-NcVolQosPolicyGroup -PolicyGroup $policyGroup
Volume Options
Setting volume options allows you to customize the volume to particular applications and uses. Let’s look at showing, getting, and setting some options.
1 2 |
# get options for a volume Get-NcVol $volumeName | Get-NcVolOption |
All of the options you would normally modify at the command line can be manipulated using PowerShell.
For most NFS volumes which are serving VMware datastores, here are the settings I use:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
$volume = Get-NcVol $volumeName # convert and create unicode, to ensure that filenames and directories are # in a unified format $volume | Set-NcVolOption -Key convert_ucode -Value on $volume | Set-NcVolOption -Key create_ucode -Value on # disable access time updates for files, which saves a few IOPS $volume | Set-NcVolOption -Key no_atime_update -Value on # do not automatically take snapshots (I rely on VSC B&R for this) $volume | Set-NcVolOption -Key nosnap -Value on # the .snapshot directory must be visible for VSC B&R to work correctly $volume | Set-NcVolOption -Key nosnapdir -Value off |
Snapshots
Snapshots are the core principle behind NetApp data protection technology. They are instant, have no performance penalty, and can be reverted to at any time quickly and easily.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
# creating a snapshot Get-NcVol $volumeName | New-NcSnapshot -Snapshot "An Example Snapshot" # listing snapshots Get-NcVol $volumeName | Get-NcSnapshot # reverting to a snapshot. BE CAREFUL! if you revert a node root volume # it will cause the node to reboot! Get-NcVol $volumeName | Restore-NcSnapshotVolume -SnapName $snapName # deleting a snapshot Remove-NcSnapshot -Volume $volumeName -Snapshot $snapName # delete all snaps for a volume Get-NcVol $volumeName | Get-NcSnapshot | Remove-NcSnapshot |
FlexClones
A FlexClone is a copy of a volume based on a snapshot that usually consumes no additional space, except for new and changed data. It is a writable instance of the data that is contained in the volume at the time of the snapshot. They are particularly useful for testing and development where you can snapshot the production data, FlexClone it to a writable volume, and then do test/dev with real, production data. Combining this with QoS creates the ability to do this without affecting performance for the production environment.
1 2 3 4 5 6 7 8 9 10 11 12 |
# create a FlexClone, this will create a new snapshot as the base Get-NcVol $volumeName | New-NcVolClone -CloneVolume $clonedVolName # create a FlexClone, using an existing snapshot as the base Get-NcVol $volumeName | New-NcVolClone -CloneVolume $clonedVolName -ParentSnapshot $snapName # split a FlexClone Get-NcVol $clonedVolName | Start-NcVolCloneSplit # check the progress of a clone split Get-NcVolCloneSplit -Name $clonedVolName |
Splitting a FlexClone is an operation that happens when you want to separate the cloned instance from the original. It is useful when you want to create a full copy of the data for any reason.
Remember that you can not delete the base snapshot of a volume with a FlexClone child until either the clone has been split, or the clone has been destroyed (it is managed just like a regular volume).
Volume Moves
Volume moves are a feature of clustered Data ONTAP (they aren’t available in 7-mode), and are a feature that fundamentally changes how storage administration is done. Using volume move operations it’s possible to vacate controllers, disk shelves, and aggregates to allow non-disruptive maintenance and lifecycle operations.
1 2 3 4 5 6 7 8 9 10 11 |
# check with aggrs a volume can move to Get-NcVol $volumeName | Get-NcVolMoveTargetAggr # initialize a volume move Get-NcVol $volumeName | Start-NcVolMove -DestinationAggregate $newAggrName # checking the status of the move Get-NcVolMove $volumeName # stopping / canceling a move Get-NcVol $volumeName | Stop-NcVolMove |
When executing Remove-NCLunMap and Set-NcVol -offline through PowerShell, you get a Yes/No/Cancel prompt box. Is there any options to force these commands to execute without prompts?
Hi Todd,
Thanks for reading!
You should be able to avoid the message by using the parameter
-Confirm:$false
with the cmdlets.Hope that helps!
Andrew
Thanks for the great article and the reply. It has been a huge help.
Thanks for taking the time to explain and post these, Andrew. Very helpful!
Hi, thanks for the great article. I am new to PS and I am trying to automate the collcetion of “space used” of a certain volume with date and time on a daily basis and write append it to csv or better xls. I managed to create the script, but not with date and time added. xls should look like this: “date”,”time”,”Volume”,”VolumeSize”,”TotalUsed”. Sounds easy, but I just cannot manage to add date and time . What I have so far ist: 1. $outfile= c:\path 2. get-navolspace -Name volumename | Select-Object Volume,VolumeSize,TotalUsed | Export-csv $outfile -NoTypeInformation -Append . Any hints?
Hi Thomas,
At the simplest you should be able to do something like this:
Hope that helps!
Andrew
Thank you for the great article! It is informative and very well laid out. I am new to Powershell and had a question. I have to create a large number of volumes. I have a file with all the variable ($volumeName, $aggrname, $volSize, Etc.). How can I use this file to create volumes using your create a volume script?
Thank you!
Paul
I wanted to know if I can list things like snapshot policy, snapvault policy and snapvault target volume name as well as snapmirro policy and info regarding that for a volume. all command I can find does not give all info regarding a volume , I want to see if I can get commands which can give volume , its snapshot, its snapvault, its snapmirror, its export policy/rule info
Hi Chen,
I have examples of how to collect that information in other posts: Volume Snapshots and Data Protection.
Once you collect the information, you’ll just need to format it however you see fit.
Hope that helps.
Andrew
@Andrew This site has been a beacon in trying to understand how to interact with the NetApp via powershell, but as others (read: mgmt) have learned of some of the capability they tasked me with trying to ‘find’ out a few things to make administration more automated like:
Enabling users to create their own volumes, qtrees, and quotas
While not crazy about it I have to see if it is actually feasible & I think the fundamental answer is yes.
This would of course all go through an API call, but would call the new-ncvol command with certain parameters (like aggregate, vservercontext, efficiencypolicy, junctionpoint=$name, etc??) already set – maybe a template would work??
But for things like volume name (junction point would equal their input name) and size they could input this information (with some limitation/restriction; still trying to figure that one out).
Is that possible & easy to do?
Honestly I think it would be better to have a default volumes and allow them to create qtrees for their projects and then set quotas that they could manage. That way i can use get-ncquota / get-ncquotareport to show them the values via JSON on a nice web interface with links to be able to manage the quota from another API.
Am I off my rocker? Is there a better way of allowing the end users to better create/manage space on the filer using the PowerShell toolkit and APIs?
I know this is a year down the track…but we have recently fired up NetApp Workflow Automation. My goodness it works a treat and takes a LOAD of work off our guys, while keeping naming conventions, replication policies and snapshot policies all set correctly. I’d definitely look at that as another option.
Hi Andrew, I am trying to put together a script to perform a volume move post volume move validation and aggregate space utilization pre-checks. Post the volume move is in progress im trying to send status of volume moves and while its done send the completion status.
Let me know if you have any ideas or thoughts that might help me give the final touch. I have made some logic and trying to get it validated with you. Just so that you are aware i wanted you to know that im not a scripting or automation expert but i can do basic scripting only 🙂
How do I use the command
Set-NcSnapshotSnaplockExpTime -volume victor_2019_test -Snapshot daily.2020-06 -Expirytime 06/14/2020
when I run it on my script I get the following error
Set-NcSnapshotSnaplockExpTime : No such file or directory
At H:\test_snaplock_etime.ps1:38 char:1
+ Set-NcSnapshotSnaplockExpTime -volume victor_2019_test -Snapshot dail …
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (durarcclu01:NcController) [Set-NcSnapshotSnaplockExpTime], EAPIERROR
+ FullyQualifiedErrorId : ApiException,DataONTAP.C.PowerShell.SDK.Cmdlets.Snapshot.SetNcSnapshotSnaplockExpTime