Monthly Archives: November 2013

Installing and Configuring the NetApp NFS Plug-in v1.0.20 for VMware VAAI

30 Saturday Nov 2013

Posted by in NetApp, VMware

≈ 2 Comments

Tags

,

The plug-in installs on the VMware ESXi v5x host. It takes advantage of vSphere’s enhanced storage features. On the NetApp controller, the nfs.vstorage.enable option has to be set to “on” so the ESXi host can take advantage of VMware VAAI.  This plug-in performs NFS-like RPCs to the server, using the same credentials as that of an ESXi NFS client. That means the plug-in needs no other permissions and has the same access rights as the ESXi NFS client. This is supported with DOT 8.1.1 and later.

The NFS plug-in includes these features:

Copy Offload – A process that used to take a few minutes now runs in seconds. This reduces traffic on the ESXi host and lowers CPU utilization for that task.

Space Reservation – This allows you to create thick virtual disks on NFS datastores. Through the VAAI Reserve Space primitive, you reserve space for the file when its created.

Download the plugin here: http://support.netapp.com/NOW/download/software/nfs_plugin_vaai/1.0.20/

1) Configure the NetApp Controller (this is for 7-Mode).
> options nfs.vstorage.enable on

2) In vCenter, select an ESXi host. Select the Configuration tab and then Storage under Hardware.
a) Under Datastores, right-click datastore1 (or whatever your local datastore is named) and select Browse datastore.
b) Click the Upload icon and select Upload a file. Browse to the NetAppNasPlugin.v20.zip file and click Open > Yes.

3) Enable SSH on the ESXi host or use the console CLI.
a) In vCenter, select the host > Configuration tab > Security Profile > across from Services, click Properties.
b) Scroll down to SSH and click Options. Click Start > OK > OK.

4) Verify that VAAI is enabled on the ESXi host. The output should be 1:
> esxcfg-advcfg -g /DataMover/HardwareAcceleratedMove
Value of HardwareAcceleratedMove is 1

> esxcfg-advcfg -g /DataMover/HardwareAcceleratedInit
Value of HardwareAcceleratedInit is 1

If VAAI is not enabled, enable it now:
> esxcfg-advcfg -s 1 /DataMover/HardwareAcceleratedInit
> esxcfg-advcfg -s 1 /DataMover/HardwareAcceleratedMove

5) Install the Plugin on the ESXi host.
> esxcli software vib install -d “/vmfs/volumes/<your path>/NetAppNasPlugin.v20.zip”
Installation Result
Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.
Reboot Required: true
VIBs Installed: NetApp_bootbank_NetAppNasPlugin_1.0-020
VIBs Removed:
VIBs Skipped:

6) Reboot the ESXi host. Either through vCenter or at the command line.
> reboot

7) Verify the plugin is installed on the ESXi host. You will have to re-enable SSH in vCenter.
> esxcli software vib get | grep -i NetApp
NetApp_bootbank_NetAppNasPlugin_1.0-020
Name: NetAppNasPlugin
Vendor: NetApp
Summary: NAS VAAI NetApp Plugin
Description: NetApp NAS VAAI Module for ESX Server
Payloads: NetAppNasPlugin

8) Create an NFS export on the NetApp Controller and mount it as a new NFS datastore on the ESXi host. These steps below are specific to my configuration but you should be able to figure out your variables.
a) On the NetApp:
> exportfs -p rw=10.10.10.0/24,root=10.10.10.0/24 /vol/vol1
(substitute your ESXi host IP range)

b) On the ESXi host:
> esxcli storage nfs add -H labnetapp01 -s /vol/vol1 -v DatastoreVAAI
(substitute your controller hostname, volume name and datatstore name)

9) Verify that the new datastore is VAAI supported with the following command. Look for NAS VAAI Supported: YES at the bottom of the output.
> vmkfstools -Ph /vmfs/volumes/<name-of-your-datastore>
NFS-1.00 file system spanning 1 partitions.
File system label (if any): DatastoreVAAI
Mode: public
Capacity 8 GB, 8 GB available, file block size 4 KB
UUID: 69e81cd6-90fa0446-0000-000000000000
Partitions spanned (on “notDCS”):
nfs:DatastoreVAAI
NAS VAAI Supported: YES
Is Native Snapshot Capable: YES

10) You can also verify VAAI support with the following commands.
> esxcli storage core plugin list
Plugin name       Plugin class
—————-  ————
VMW_VAAIP_NETAPP  VAAI
VAAI_FILTER       Filter
NMP               MP

> esxcli storage core claimrule list –claimrule-class=VAAI | grep NETAPP
VAAI        65433  runtime  vendor  VMW_VAAIP_NETAPP  vendor=NETAPP model=*
VAAI        65433  file     vendor  VMW_VAAIP_NETAPP  vendor=NETAPP model=*

> esxcli storage core claimrule list –claimrule-class=Filter | grep NETAPP
Filter      65433  runtime  vendor  VAAI_FILTER  vendor=NETAPP model=*
Filter      65433  file     vendor  VAAI_FILTER  vendor=NETAPP model=*

HOWTO Secure iSCSI Luns Between VMware ESXi 5.1 and NetApp Storage with Bidirectional CHAP

29 Friday Nov 2013

Posted by in iSCSI, Security, VMware

≈ Leave a comment

Tags

, ,

This document demonstrates how to configure iSCSI bidirectional CHAP between ESXi 5.1 Update 1 and NetApp storage. There were to many screen captures for a regular post so I created the pdf below.

Versions in use for this document:
1) VMware vSphere vCenter 5.1 Update 1c Web Client
2) VMware ESXi 5.1 Update 1
a. Note: The ESXi software iSCSI initiator will be used for this document
3) NetApp Data ONTAP 7.3.7P3 (this also works with Data ONTAP version 8x)

The HOWTO is here: Bidirectional CHAP with vSphere 5.1u1c.pdf

When a VMware Tools Upgrade Goes Bad – Fixing NICs

27 Wednesday Nov 2013

Posted by in VMware, Windows

≈ Leave a comment

Tags

,

Scenario: you upgrade VMware tools on one of your VMs and your IP address will not maintain a static configuration. It reverts back to a Microsoft APIPA address (169.254.0.1 – 169.254.255.254). It’s Sunday at 9:00 PM, the outage window is closing and now you are angry.

Solution: you have to clean out all references to current and previous NICs in the registry. This post tells you how to do this for Windows 2008R2 x64. Make sure you have a local administrator account and know the password before you start. You don’t want to lock yourself out of the VM should something go haywire.

Note: Make sure your VM hardware is set to a version compatible with your version of vSphere. For this post, I’m set to version 9 (for 5.1 U1c). You can run into issues related to buggy hardware mismatches.

1) Remove the NIC and reboot the VM.
a) Right-click the VM > Edit Settings > select the NIC and click Remove > OK.

2) Delete the Interfaces and Adapters from the Registry.
a) Click Start > Run and enter regedt32 and press enter.
b) Navigate to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\Tcpip\Parameters\Interfaces
c) Expand Interfaces and delete all entries (delete folders from the left pane).
d) Navigate to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\Tcpip\Parameters\Adapters
e) Expand Adapters and delete all entries (delete folders from the left pane).
f) Close the registry editor and reboot the VM.

3) Add the VMXNET3 NIC back to the VM.
a) Right-click the VM > Edit Settings > click Add > select Ethernet Adapter and click Next.
b) On the Network Connection page, select the VMXNET3 adapter type, select the network you want to connect to and most importantly, DESELECT connect at power on, and then click Next > Finish > OK.
c) Reboot the VM. Don’t skip this reboot.

4) Configure your IP address.
a) Login to the VM. Click Start > Control Panel > Network and Internet > Network and Sharing Center > Change Adapter Setting.
b) Right-click the NIC > Properties > select IPv4 and click Properties.
c) Enter your IP, mask, gateway, dns and click OK > OK.
d) Right-click the VM > Edit Settings > select the NIC and in the upper right, click both Connect at power on and Connected, then click OK.
e) The NIC will connect and you should have a clean network configuration.
f) Open a cmd prompt and enter ipconfig -all to verify. Ping other hosts to test.

HOWTO Secure iSCSI Luns Between CentOS Linux 5.10 and NetApp Storage with Mutual CHAP

17 Sunday Nov 2013

Posted by in iSCSI, Linux, Security

≈ Leave a comment

Tags

, ,

This post demonstrates how to enable two-way or mutual CHAP on iSCSI luns between CentOS Linux 5.10 and NetApp storage. The aggregate, lun and disk sizes are small in this HOWTO to keep it simple.

1) Install open-iscsi on your server.
> yum install iscsi-initiator*
> reboot (don’t argue with me, just do it!)

2) Display your server’s new iscsi initiator or iqn nodename.
> cat /etc/iscsi/initiatorname.iscsi
InitiatorName=iqn.1994-05.com.redhat:01241a79a24

3) On the NetApp filer, create the volume that will hold the iscsi luns. This command assumes you have aggregate aggr1 already created. If not use an aggregate that has enough room for your volume.
netapp> vol create MCHAPVOL aggr1 10g

4) Create the lun in the volume.
netapp> lun create -s 5g -t linux /vol/MCHAPVOL/CENTSOS510_iSCSI_MCHAP_01

5) Create an igroup and add the Linux iscsi nodename or iqn from step 2 above to it.
netapp> igroup create -i -t linux ISCSI_MCHAP_CENTOS510
netapp> igroup add ISCSI_MCHAP_CENTOS510 iqn.1994-05.com.redhat:01241a79a24
netapp> igroup show ISCSI_MCHAP_CENTOS510

ISCSI_MCHAP_CENTOS510 (iSCSI) (ostype: linux):
iqn.1994-05.com.redhat:01241a79a24 (not logged in)

6) Map the lun to the igroup and give it lun ID 01.
netapp> lun map /vol/MCHAPVOL/CENTSOS510_iSCSI_MCHAP_01 ISCSI_MCHAP_CENTOS510 01

7) Obtain the NetApp target nodename.
netapp> iscsi nodename
iqn.1992-08.com.netapp:sn.84167939

8) Set the CHAP secret on the NetApp controller.
netapp> iscsi security add -i iqn.1994-05.com.redhat:01241a79a24 -s chap -p MCHAPCENT510 -n iqn.1994-05.com.redhat:01241a79a24 -o NETAPPMCHAP -m iqn.1992-08.com.netapp:sn.84167939

netapp> iscsi security show
init: iqn.1994-05.com.redhat:01241a79a24 auth: CHAP Inbound password: **** Inbound username: iqn.1994-05.com.redhat:01241a79a24 Outbound password: **** Outbound username: iqn.1992-08.com.netapp:sn.84167939

9) On the server, edit your /etc/iscsi/iscsi.conf file and set the parameters below.  
> vi /etc/iscsi/iscsid.conf:
node.startup = automatic
node.session.auth.authmethod = CHAP
node.session.auth.username = iqn.1994-05.com.redhat:01241a79a24
node.session.auth.password = MCHAPCENT510
node.session.auth.username_in = iqn.1992-08.com.netapp:sn.84167939
node.session.auth.password_in = NETAPPMCHAP
discovery.sendtargets.auth.authmethod = CHAP
discovery.sendtargets.auth.username = iqn.1994-05.com.redhat:01241a79a24
discovery.sendtargets.auth.password = MCHAPCENT510
discovery.sendtargets.auth.username_in = iqn.1992-08.com.netapp:sn.84167939
discovery.sendtargets.auth.password_in = NETAPPMCHAP
> wq!

10) On the server, restart the service and discover your iSCSI target (your storage system).
> service iscsi restart
> iscsiadm -m discovery -t st -p 10.10.10.11
10.10.10.11:3260,1000 iqn.1992-08.com.netapp:sn.84167939

> iscsiadm -m node  (this should display the same as above)
10.10.10.11:3260,1000 iqn.1992-08.com.netapp:sn.84167939

11) On the server, manually login to the iSCSI target (your storage array). Note there are two dashes “- -” in front of targetname and login.
> iscsiadm -m node –targetname “iqn.1992-08.com.netapp:sn.84167939” –login

Logging in to [iface: default, target: iqn.1992-08.com.netapp:sn.84167939, portal: 10.10.10.11,3260] (multiple)
Login to [iface: default, target: iqn.1992-08.com.netapp:sn.84167939, portal: 10.10.10.11,3260] successful.

On the NetApp storage console you should see the iSCSI sessions:
Sun Nov 17 07:51:00 EST [iscsi.notice:notice]: ISCSI: New session from initiator iqn.1994-05.com.redhat:01241a79a24  at IP addr 10.10.10.37
Sun Nov 17 07:56:38 EST [iscsi.notice:notice]: ISCSI: New session from initiator iqn.1994-05.com.redhat:01241a79a24 at IP addr 10.10.10.37

Verify the iSCSI session on the filer:
netapp> iscsi session show
Session 5
Initiator Information
Initiator Name: iqn.1994-05.com.redhat:01241a79a24
ISID: 00:02:3d:01:00:00
Initiator Alias: cent510

12) Stop and start the iscsi service on the server.
> service iscsi stop
Pause for 10 seconds and then run the next command.
> service iscsi start

13) From the server , check your session.
> iscsiadm -m session -P 1
Target: iqn.1992-08.com.netapp:sn.84167939
Current Portal: 10.10.10.11:3260,1000
Persistent Portal: 10.10.10.11:3260,1000
**********
Interface:
**********
Iface Name: default
Iface Transport: tcp
Iface Initiatorname: iqn.1994-05.com.redhat:01241a79a24
Iface IPaddress: 10.10.10.37
Iface HWaddress: <empty>
Iface Netdev: <empty>
SID: 2
iSCSI Connection State: LOGGED IN
iSCSI Session State: LOGGED_IN
Internal iscsid Session State: NO CHANGE

14) From the server, check the NetApp iSCSI details. Note there are two dashes “- -” in front of mode, targetname and portal.
> iscsiadm -–mode node –-targetname “iqn.1992-08.com.netapp:sn.84167939” –-portal 10.10.10.11:3260

15) From the server, find and format the new lun (new disk).
> cat /var/log/messages | grep “unknown partition table”
Nov 17 04:56:52 cent510 kernel:  sdb: unknown partition table

> fdisk /dev/sdb

Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel. Changes will remain in memory only,
until you decide to write them. After that, of course, the previous
content won’t be recoverable.

Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

> fdisk /dev/sdb
Command (m for help): n
Command action
e   extended
p   primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-1018, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-1018, default 1018):
Using default value 1018

Command (m for help): p

Disk /dev/sdb: 5368 MB, 5368709120 bytes
166 heads, 62 sectors/track, 1018 cylinders
Units = cylinders of 10292 * 512 = 5269504 bytes

Device Boot      Start         End      Blocks   Id  System
/dev/sdb1               1        1018     5238597   83  Linux

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

16) On the server, create the Linux file system on the new partition.
> mkfs -t ext3 /dev/sdb1
mke2fs 1.39 (29-May-2006)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
655360 inodes, 1309649 blocks
65482 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=1342177280
40 block groups
32768 blocks per group, 32768 fragments per group
16384 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736

Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 26 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.

17) Verify the partition.
> blkid /dev/sdb1
/dev/sdb1: UUID=”90e6dc7f-13ac-42bd-b471-b6f8e6863414″ SEC_TYPE=”ext2″ TYPE=”ext3″

18) Create the mount point and manually mount the directory.
> mkdir /newiscsilun
> mount /dev/sdb1 /newiscsilun
> df -h | grep newiscsilun
Filesystem Size  Used Avail Use% Mounted on
/dev/sdb1  5.0G  139M  4.6G   3% /newiscsilun

19) Add the new mount point to /etc/fstab.
> vi /etc/fstab
/dev/sdb1 /newiscsilun ext4 _netdev 0 0
> wq!

Note: the _netdev option is important so that it doesn’t try mounting the target before the network is available.

20) Test that it survives a reboot by rebooting the server. With the _netdev set, iscsi starts and your CHAP logins should take place before it attempts to mount. After the reboot, login and verify its mounted.

> df -h | grep newiscsilun
Filesystem Size  Used Avail Use% Mounted on
/dev/sdb1  5.0G  139M  4.6G   3% /newiscsilun

21) On the server you can check session stats.
> iscsiadm -m session -s
Stats for session [sid: 1, target: iqn.1992-08.com.netapp:sn.84167939, portal: 10.10.10.11,3260]
iSCSI SNMP:
txdata_octets: 22136
rxdata_octets: 377532
noptx_pdus: 0
scsicmd_pdus: 60
tmfcmd_pdus: 0
login_pdus: 0
text_pdus: 0
dataout_pdus: 0
logout_pdus: 0
snack_pdus: 0
noprx_pdus: 0
scsirsp_pdus: 60
tmfrsp_pdus: 0
textrsp_pdus: 0
datain_pdus: 56
logoutrsp_pdus: 0
r2t_pdus: 0
async_pdus: 0
rjt_pdus: 0
digest_err: 0
timeout_err: 0
iSCSI Extended:
tx_sendpage_failures: 0
rx_discontiguous_hdr: 0
eh_abort_cnt: 0

22) As root, change permissions on /etc/iscsi/iscsid.conf. I’m not sure why they haven’t fixed this clear text CHAP password in a file issue so just make sure only root can read/write the file.
> chmod 600 /etc/iscsi/iscsid.conf

23) On the NetApp storage you can verify the Lun and the server’s session.
> lun show -v /vol/MCHAPVOL/CENTSOS510_iSCSI_MCHAP_01
/vol/MCHAPVOL/CENTSOS510_iSCSI_MCHAP_01      5g (5368709120)    (r/w, online, mapped)
Serial#: hoagPJupukXK
Share: none
Space Reservation: enabled
Multiprotocol Type: linux
Maps: ISCSI_MCHAP_CENTOS510=1

>  iscsi session show -v
Session 7
Initiator Information
Initiator Name: iqn.1994-05.com.redhat:01241a79a24
ISID: 00:02:3d:01:00:00
Initiator Alias: cent510

Session Parameters
SessionType=Normal
TargetPortalGroupTag=1000
MaxConnections=1
ErrorRecoveryLevel=0
AuthMethod=CHAP
HeaderDigest=None
DataDigest=None
ImmediateData=Yes
InitialR2T=No
FirstBurstLength=65536
MaxBurstLength=65536
Initiator MaxRecvDataSegmentLength=65536
Target MaxRecvDataSegmentLength=65536
DefaultTime2Wait=2
DefaultTime2Retain=0
MaxOutstandingR2T=1
DataPDUInOrder=Yes
DataSequenceInOrder=Yes
Command Window Size: 32

Connection Information
Connection 0
Remote Endpoint: 10.10.10.37:44786
Local Endpoint: 10.10.10.11:3260
Local Interface: e0a
TCP recv window size: 131400

Command Information
No commands active

HOWTO use Wireshark to read a packet capture from NetApp Data ONTAP after running the pktt command.

08 Friday Nov 2013

Posted by in NetApp, Wireshark

≈ 1 Comment

Tags

,

NetApp Data ONTAP 7 and 8 has the ability to sniff packets but the trace file cant be viewed on the controller. You can open and manipulate the trace file in Wireshark on another host. This HOWTO uses Wireshark on Windows 7. Wireshark on Linux will work as well. You must have Wireshark already installed on your Windows/Linux host before you start. You can download it here:

Windows: http://www.wireshark.org/download.html

Debian based Linux:
> apt-get install wireshark

RPM based Linux:
> yum install wireshark

1) Identify the controller’s NIC where you want to sniff packets on. In this case we will use e0a.
netapp> ifconfig -a

e0a: flags=0xe48867<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.10.10.140 netmask 0xffffff00 broadcast 10.10.10.255
ether 00:0c:29:89:3f:3c (auto-1000t-fd-up) flowcontrol full
e0b: flags=0xe08866<BROADCAST,RUNNING,MULTICAST> mtu 1500
ether 00:0c:29:89:3f:46 (auto-1000t-fd-up) flowcontrol full
e0c: flags=0xe08866<BROADCAST,RUNNING,MULTICAST> mtu 1500
ether 00:0c:29:89:3f:50 (auto-1000t-fd-up) flowcontrol full
e0d: flags=0xe08866<BROADCAST,RUNNING,MULTICAST> mtu 1500
ether 00:0c:29:89:3f:5a (auto-1000t-fd-up) flowcontrol full
lo: flags=0x1b48049<UP,LOOPBACK,RUNNING,MULTICAST,TCPCKSUM> mtu 9188
inet 127.0.0.1 netmask 0xff000000 broadcast 127.0.0.1
losk: flags=0x40a400c9<UP,LOOPBACK,RUNNING> mtu 9188
inet 127.0.20.1 netmask 0xff000000 broadcast 127.0.20.1

2) Using the pktt command, start the capture on interface e0a and dump the output into /etc/log on the controller. When you run the command, a file is created in /etc/log/ with the NIC name (e0a), a date/time stamp and a .trc file extension.
netapp> pktt start e0a -d /etc/log
e0a: started packet trace

3) You can check the status of the packet capture and get details.
netapp> pktt status
e0a: Packet tracing enabled; packets truncated at 1514 bytes.
e0a: Trace buffer utilization = 2% of 1048320 bytes, 258 packets
e0a: 0 bytes written to file /etc/log/e0a_20131108_173928.trc
e0a: Currently tracing to file /etc/log/e0a_20131108_173928.trc
e0a: 258 packets seen; 0 packets dropped; 24936 total bytes seen

lo: Packet tracing enabled; packets truncated at 1514 bytes.
lo: Trace buffer utilization = 99% of 130816 bytes, 1011 packets
lo: 1387 packets seen; 0 packets dropped; 160568 total bytes seen

losk: Packet tracing enabled; packets truncated at 1514 bytes.
losk: Trace buffer utilization = 99% of 130816 bytes, 282 packets
losk: 40901 packets seen; 0 packets dropped; 21761277 total bytes seen

4) After a period of time you deem adequate, stop the packet capture.
netapp> pktt stop e0a
e0a: Tracing stopped and packet trace buffers released.
Fri Nov  8 17:42:25 EST [sim81:cmds.pktt.write.info:info]: pktt: 280 packets seen, 0 dropped, 32046 bytes written to /etc/log/e0a_20131108_173928.trc.

5) Verify that it has stopped.
netapp> pktt status
e0a: packet tracing not enabled

6) Open Windows Explorer on the PC/Server and enter the UNC path to the /etc/ folder on the filer. If you don’t have CIFS enabled and use NFS, mount the file system to your UNIX host.   \\10.10.10.140\etc$

pktt01

7) Browse to the log folder and locate the .trc file you just created. Double-click the file and it will load in Wireshark.

pktt02

8) You can now operate on the trace file and filter, search and analyze packets.

pktt03