This post demonstrates how to enable bidirectional or mutual CHAP on iSCSI luns between Oracle Enterprise Linux 6 update 5 and NetApp storage. The aggregate, lun and disk sizes are small in this HOWTO to keep it simple.
1) Install open-iscsi on your server.
> yum install iscsi-initiator*
> reboot (don’t argue with me, just do it!)
2) Display your server’s new iscsi initiator or iqn nodename.
> cat /etc/iscsi/initiatorname.iscsi
InitiatorName=iqn.1988-12.com.oracle:523325af23
3) On the NetApp filer, create the volume that will hold the iscsi luns. This command assumes you have aggregate aggr1 already created. If not, use an aggregate that has enough room for your volume.
netapp> vol create MCHAPVOL aggr1 10g
4) Create the lun in the volume.
netapp> lun create -s 5g -t linux /vol/MCHAPVOL/OEL6u5_iSCSI_MCHAP_01
5) Create an igroup and add the Oracle Enterprise Linux iscsi nodename or iqn from step 2 above to it.
netapp> igroup create -i -t linux ISCSI_MCHAP_OEL6u5
netapp> igroup add ISCSI_MCHAP_OEL6u5 iqn.1988-12.com.oracle:523325af23
netapp> igroup show ISCSI_MCHAP_OEL6u5
ISCSI_MCHAP_OEL6u5 (iSCSI) (ostype: linux):
iqn.1988-12.com.oracle:523325af23 (not logged in)
6) Map the lun to the igroup and give it lun ID 01.
netapp> lun map /vol/MCHAPVOL/OEL6u5_iSCSI_MCHAP_01 ISCSI_MCHAP_OEL6u5 01
7) Obtain the NetApp target nodename.
netapp> iscsi nodename
iqn.1992-08.com.netapp:sn.84167939
8) Set the CHAP secret on the NetApp controller.
netapp> iscsi security add -i iqn.1988-12.com.oracle:523325af23 -s chap -p MCHAPOEL6u5 -n iqn.1988-12.com.oracle:523325af23 -o NETAPPMCHAP -m iqn.1992-08.com.netapp:sn.84167939
netapp> iscsi security show
init: iqn.1988-12.com.oracle:523325af23 auth: CHAP Inbound password: **** Inbound username: iqn.1988-12.com.oracle:523325af23 Outbound password: **** Outbound username: iqn.1992-08.com.netapp:sn.84167939
9) On the server, edit your /etc/iscsi/iscsi.conf file and set the parameters below.
> vi /etc/iscsi/iscsid.conf
node.startup = automatic
node.session.auth.authmethod = CHAP
node.session.auth.username = iqn.1988-12.com.oracle:523325af23
node.session.auth.password = MCHAPOEL6u5
node.session.auth.username_in = iqn.1992-08.com.netapp:sn.84167939
node.session.auth.password_in = NETAPPMCHAP
discovery.sendtargets.auth.authmethod = CHAP
discovery.sendtargets.auth.username = iqn.1988-12.com.oracle:523325af23
discovery.sendtargets.auth.password = MCHAPOEL6u5
discovery.sendtargets.auth.username_in = iqn.1992-08.com.netapp:sn.84167939
discovery.sendtargets.auth.password_in = NETAPPMCHAP
> wq!
10) On the server, restart the service and discover your iSCSI target (your storage system).
> service iscsi restart
> iscsiadm -m discovery -t st -p 10.10.10.11
10.10.10.11:3260,1000 iqn.1992-08.com.netapp:sn.84167939
> iscsiadm -m node (this should display the same as above)
10.10.10.11:3260,1000 iqn.1992-08.com.netapp:sn.84167939
11) On the server, manually login to the iSCSI target (your storage array). Note there are two dashes “- -” in front of –login. It always looks like one.
> iscsiadm -m node -T “iqn.1992-08.com.netapp:sn.84167939” –login
Logging in to [iface: default, target: iqn.1992-08.com.netapp:sn.84167939, portal: 10.10.10.11,3260] (multiple)
Login to [iface: default, target: iqn.1992-08.com.netapp:sn.84167939, portal: 10.10.10.11,3260] successful.
Verify the iSCSI session on the filer:
netapp> iscsi session show
Session 10
Initiator Information
Initiator Name: iqn.1988-12.com.oracle:523325af23
ISID: 00:02:3d:01:00:00
Initiator Alias: oel6u5
12) Stop and start the iscsi service on the server.
> service iscsi stop
Pause for 10 seconds and then run the next command.
> service iscsi start
13) From the server, check your session.
> iscsiadm -m session -P 1
Target: iqn.1992-08.com.netapp:sn.84167939
Current Portal: 10.10.10.11:3260,1000
Persistent Portal: 10.10.10.11:3260,1000
**********
Interface:
**********
Iface Name: default
Iface Transport: tcp
Iface Initiatorname: iqn.1988-12.com.oracle:523325af23
Iface IPaddress: 10.10.10.93
Iface HWaddress: <empty>
Iface Netdev: <empty>
SID: 2
iSCSI Connection State: LOGGED IN
iSCSI Session State: LOGGED_IN
Internal iscsid Session State: NO CHANGE
14) From the server, check the NetApp iSCSI details. Note there are two dashes “- -” in front of mode, targetname and portal. Sometimes it looks like one.
> iscsiadm -–mode node –-targetname “iqn.1992-08.com.netapp:sn.84167939″ –-portal 10.10.10.11:3260
# BEGIN RECORD 6.2.0-873.10.el6
node.name = iqn.1992-08.com.netapp:sn.84167939
node.tpgt = 1000
node.startup = automatic
node.leading_login = No
iface.hwaddress = <empty>
iface.ipaddress = <empty>
iface.iscsi_ifacename = default
<output truncated to keep the post short>
15) From the server, find and format the new lun (new disk). On the fdisk command wizard, enter the letters in bold below.
> cat /var/log/messages | grep “unknown partition table”
Dec 14 08:55:02 oel6u5 kernel: sdb: unknown partition table
> fdisk /dev/sdb
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel with disk identifier 0x54ac8aa4.
Changes will remain in memory only, until you decide to write them.
After that, of course, the previous content won’t be recoverable.
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)
WARNING: DOS-compatible mode is deprecated. It’s strongly recommended to
switch off the mode (command ‘c’) and change display units to
sectors (command ‘u’).
Command (m for help): u
Changing display/entry units to sectors
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
> fdisk /dev/sdb
WARNING: DOS-compatible mode is deprecated. It’s strongly recommended to
switch off the mode (command ‘c’) and change display units to
sectors (command ‘u’).
Command (m for help): c
DOS Compatibility flag is not set
Command (m for help): u
Changing display/entry units to sectors
Command (m for help): n
Command action
e extended
p primary partition (1-4) <press the P key>
p
Partition number (1-4): 1
First sector (2048-10485759, default 2048): <press enter>
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-10485759, default 10485759): <press enter>
Using default value 10485759
Command (m for help): p
Disk /dev/sdb: 5368 MB, 5368709120 bytes
166 heads, 62 sectors/track, 1018 cylinders, total 10485760 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x54ac8aa4
Device Boot Start End Blocks Id System
/dev/sdb1 2048 10485759 5241856 83 Linux
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
16) On the server, create the Linux file system on the new partition.
> mkfs -t ext4 /dev/sdb1
mke2fs 1.43-WIP (20-Jun-2013)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
327680 inodes, 1310464 blocks
65523 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=1342177280
40 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736
Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done
17) Verify the partition.
> blkid /dev/sdb1
/dev/sdb1: UUID=”1a6e2a56-924f-4e3b-b281-ded3a3141ab4″ TYPE=”ext4″
18) Create the mount point and manually mount the directory.
> mkdir /newiscsilun
> mount /dev/sdb1 /newiscsilun
> df -h | grep newiscsilun
Filesystem Size Used Avail Use% Mounted on
/dev/sdb1 4.8G 10M 4.6G 1% /newiscsilun
19) Add the new mount point to /etc/fstab.
> vi /etc/fstab
/dev/sdb1 /newiscsilun ext4 _netdev 0 0
> wq!
Note: the _netdev option is important so that it doesn’t try mounting the target before the network is available.
20) Test that it survives a reboot by rebooting the server. With the _netdev set, iscsi starts and your CHAP logins should take place before it attempts to mount. After the reboot, login and verify its mounted.
> reboot
When done rebooting, login and verify the lun is mounted.
> df -h | grep newiscsilun
Filesystem Size Used Avail Use% Mounted on
/dev/sdb1 4.8G 10M 4.6G 1% /newiscsilun
21) On the server you can check session stats.
> iscsiadm -m session -s
Stats for session [sid: 1, target: iqn.1992-08.com.netapp:sn.84167939, portal: 10.10.10.11,3260]
iSCSI SNMP:
txdata_octets: 31204
rxdata_octets: 917992
noptx_pdus: 0
scsicmd_pdus: 270
tmfcmd_pdus: 0
login_pdus: 0
text_pdus: 0
dataout_pdus: 0
logout_pdus: 0
snack_pdus: 0
noprx_pdus: 0
scsirsp_pdus: 270
tmfrsp_pdus: 0
textrsp_pdus: 0
datain_pdus: 242
logoutrsp_pdus: 0
r2t_pdus: 0
async_pdus: 0
rjt_pdus: 0
digest_err: 0
timeout_err: 0
iSCSI Extended:
tx_sendpage_failures: 0
rx_discontiguous_hdr: 0
eh_abort_cnt: 0
22) As root, change permissions on /etc/iscsi/iscsid.conf. I’m not sure why they haven’t fixed this clear text CHAP password in a file issue so just make sure only root can read/write the file.
> chmod 600 /etc/iscsi/iscsid.conf
23) On the NetApp storage you can verify the Lun and the server’s session.
> lun show -v /vol/MCHAPVOL/OEL6u5_iSCSI_MCHAP_01
/vol/MCHAPVOL/OEL6u5_iSCSI_MCHAP_01 5g (5368709120) (r/w, online, mapped)
Serial#: hoagPJvLcRy6
Share: none
Space Reservation: enabled (not honored by containing Aggregate)
Multiprotocol Type: linux
Maps: ISCSI_MCHAP_OEL6u5=1
> iscsi session show -v
Session 12
Initiator Information
Initiator Name: iqn.1988-12.com.oracle:523325af23
ISID: 00:02:3d:01:00:00
Initiator Alias: oel6u5
Session Parameters
SessionType=Normal
TargetPortalGroupTag=1000
MaxConnections=1
ErrorRecoveryLevel=0
AuthMethod=CHAP
HeaderDigest=None
DataDigest=None
ImmediateData=Yes
InitialR2T=No
FirstBurstLength=65536
MaxBurstLength=65536
Initiator MaxRecvDataSegmentLength=65536
Target MaxRecvDataSegmentLength=65536
DefaultTime2Wait=2
DefaultTime2Retain=0
MaxOutstandingR2T=1
DataPDUInOrder=Yes
DataSequenceInOrder=Yes
Command Window Size: 32
Connection Information
Connection 0
Remote Endpoint: 10.10.10.93:33454
Local Endpoint: 10.10.10.11:3260
Local Interface: e0a
TCP recv window size: 131400
Command Information
No commands active