InfiniBand (IB) is a new I/O technology based on switch fabrics introduced in the Solaris 10 release. It provides high bandwidth, low latency interconnect for attaching I/O devices to hosts and for host-to-host communication.This is a list of the overview information in this chapter.Overview of InfiniBand Devices Dynamically Reconfiguring IB Devices (cfgadm) For information on the procedures associated with using IB devices, see the following:Dynamically Reconfiguring IB Devices (Task Map) Using the uDAPL Application Interface With InfiniBand Devices For general information about dynamic reconfiguration and hot-plugging, see Chapter 6, Dynamically Configuring Devices (Tasks). IB devices are managed by the Solaris IB nexus driver. This driver supports 5 types of devices:IB Port devices IB virtual physical point of attachment (VPPA) devices IB HCA service (HCA_SVC) devices Pseudo devices I/O controller (IOC) devices The IB nexus driver queries the Solaris IB Device Manager (IBDM) for services, referred in this guide as communication services, to enumerate the IB Port, HCA_SVC, and IB VPPA devices.The Port devices bind a communication service to a given port# of a Host Channel Adapter (HCA). The VPPA devices bind a communication service to a port#, p_key# combination instead. The HCA_SVC devices bind a communication service to a given HCA. Note that the Port devices and the HCA_SVC devices always use a p_key (partition key) whose value is zero. The Port, HCA_SVC, and VPPA devices are children of the HCA and are enumerated through the ib.conf file. For more information, see ib7D.The IOC devices are children of the IB nexus driver and are part of an I/O unit. The pseudo devices are also children of the IB nexus driver and refer to all other devices that provide their own configuration files to enumerate. For more information, see ib4. The possible IB device tree path name(s) are listed in the following table.IOC device /ib/ioc@1730000007F510C,173000007F50 IB pseudo device /ib/<driver>@<unit-address> IB VPPA device /pci@1f,2000/pci@1/pci15b3,5a44@0/ibport@<port#>,<p_key>,<service> IB HCA_SVC device /pci@1f,2000/pci@1/pci15bc,5a44@0/ibport@0,0,<service> IB Port device /pci@1f,2000/pci@1/pci15b3,5a44@0/ibport@<port#>,0,<service> HCA /pci@1f,2000/pci@1/pci15b3,5a44@0 Note that the IB HCA_SVC devices have zero as the port# and the p_key.The IB components in the preceding table are described as follows:<services>Is a communication service. For example, ipib is the communication service used by the ibd kernel client driver. <p_key>Is the partition key value being used. <port>Is the port number. <unit-address>Refers to IB kernel client driver's property by this name specified in its driver.conf file. For more information, see driver.conf4. Task Description For Instructions Display IB device information. Display information about the IB devices on your system. How to Display IB Device Information Configure or unconfigure an IOC device. Select one of the following: Unconfigure an IOC device. How to Unconfigure an IOC Device Configure an IOC device. How to Configure an IOC Device Configure or unconfigure a port or VPPA device. Select one of the following: Unconfigure a port or a VPPA device. How to Unconfigure an IB Port, HCA_SVC, or a VPPA Device Configure a port or a VPPA device. How to Configure a IB Port, HCA_SVC, or a VPPA Device Configure or unconfigure an IB pseudo device. Select one of the following: Unconfigure an IB pseudo device. How to Unconfigure an IB Pseudo Device Configure an IB pseudo device. How to Configure an IB Pseudo Device Display kernel IB clients of an HCA. You might need to display information about kernel IP clients of an HCA, particularly if you're going to unconfigure an HCA. How to Display Kernel IB Clients of an HCA Configure or unconfigure an IB HCA. Select one of the following: Unconfigure IB devices that are connected to an HCA. How to Unconfigure IB Devices Connected to an HCA Configure IB devices that are connected to an HCA. Configuring an IB HCA Update the IB p_key tables. If the p_key table information of a HCA port changes, IBTF and IBDM need to be notified so that their internal p_key databases are updated. How to Update the IB p_key Tables Display IB communication services Display the IB communication services that are currently in use by the IBTF. How to Display IB Communication Services Add or remove a VPPA communication service. Select one of the following: Add a VPPA communication service. How to Add a VPPA Communication Service Remove a VPPA communication service. How to Remove an Existing IB Port, HCA_SVC, or a VPPA Communication Service Update an IOC configuration. You can update the properties of all the IOC device nodes or update a particular IOC Ap_Id. How to Update an IOC Configuration One can configure or unconfigure an IB device from a running system by using the cfgadm CLI only. This command also provides a way to display the IB fabric, manage communication services, and update p_key table databases. For more information, see cfgadm_ib1M.The cfgadm CLI manages dynamic reconfiguration, referred to in this guide as DR, of the entire IB fabric as seen by a host. The cfgadm operations are supported on all the IB devices, such as Port, VPPA, HCA_SVC, IOC, and pseudo devices.The cfgadm command displays information about attachment points (Ap_Ids), which are locations in the system where DR operations can occur. For details on the Ap_Ids that cfgadm supports, see cfgadm_ib.1M. Note that all IB Ap_Ids are shown as connected.The cfgadm command provides the following IB device status information.Receptacle State Description connected/configured/ok The device is connected and available. The devinfo node is present. connected/unconfigured/unknown The device is unavailable and no devinfo node or device driver exists for this device. Or, the device was never configured for use by ib nexus driver. The device might be known to the IB Device Manager. The following sections describe how to dynamically reconfigure (DR) IB devices with the cfgadm command. All of the sample IB device information in these sections has been truncated to focus on relevant information.You can use the prtconf command to display general information about IB devices. For example:$ prtconf pci, instance #0 pci15b3,5a44, instance #0 ibport, instance #253 ibport, instance #254 ibport, instance #255 . . . ib, instance #0 ioc, instance #243 ioc, instance #244 ioc, instance #245 ioc, instance #246 ioc, instance #247 ioc, instance #248 ibgen, instance #249In the preceding example, pci15b3,5a44 refers to an IB HCA.Use the following steps to display specific IB device information. Become superuser. Display IB fabric information.For example:# cfgadm -a Ap_Id Type Receptacle Occupant Condition ib IB-Fabric connected configured ok hca:1730000008070 IB-HCA connected configured ok ib::1730000007F5198 IB-IOC connected configured ok ib::1730000007F5199 IB-IOC connected configured ok ib::1730000008070,0,hnfs IB-HCA_SVC connected configured ok ib::1730000008071,0,sdp IB-PORT connected configured ok ib::1730000008072,0,sdp IB-PORT connected configured ok ib::1730000008071,8001,ipib IB-VPPA connected configured ok ib::1730000008072,8001,ipib IB-VPPA connected configured ok ib::ibgen,0 IB-PSEUDO connected configured ok #In the preceding example output, the components are described as follows:Ap_Id ib::1730000008072,0,sdpIdentifies an IB Port device that is connected to port 2 and is bound to the sdp service. Ap_Id ib::1730000008072,8001,ipibIdentifies an IB VPPA device that is connected to port 2, using a p_key value of 0x8001, and is bound to the ibd service. Ap_Id ib:: 1730000008070,0,hnfsIdentifies an IB HCA_SVC device bound to the hnfs service. Ap_Id ib::1730000007F5198Identifies an IOC device. Ap_Id ib::ibgen,0Identifies a pseudo device. Display specific IB device information.For example, for an IB VPPA device:# cfgadm -al -s "cols=ap_id:info" ib::1730000008072,8001,ipib Ap_Id Information ib::1730000008072,8001,ipib ipib # For example, for an IB HCA device:# cfgadm -al -s "cols=ap_id:info" hca::1730000008070 Ap_Id Information hca::1730000008070 VID: 0x15b3, PID: 0x5a44, #ports: 0x2, port1 GUID: 0x1730000008071, port2 GUID: 0x1730000008072 # The preceding output displays the number of ports and their GUIDs. You can unconfigure an IB device that is still physically connected to the system, but a driver will never attach to it. Become superuser. Unconfigure the IB device.For example:# cfgadm -c unconfigure ib::1730000007F5198 Unconfigure the device: /devices/ib:fabric::1730000007F5198 This operation will suspend activity on the IB device Continue (yes/no)? y # Verify that the device is unconfigured.For example:# cfgadm -a ib::1730000007F5198 ib::1730000007F5198 IB-IOC connected unconfigured unknown # Become superuser. Configure a IB device.For example:# cfgadm -yc configure ib::1730000007F5198 Verify that the IB device is configured.For example:# cfgadm -al ib::1730000007F5198 Ap_Id Type Receptacle Occupant Condition ib::1730000007F5198 IB-IOC connected configured ok Use the following steps if you want to remove an IB Port, HCA_SVC, or a VPPA device from the system.The example below illustrates how to unconfigure a VPPA device, but the same procedure applies to Port and HCA_SVC devices as well. Become superuser. Unconfigure the IB VPPA device.For example:# cfgadm -c unconfigure ib::1730000007F51,8001,ipib Unconfigure the device: /devices/ib:fabric::1730000007F51,8001,ipib This operation will suspend activity on the IB device Continue (yes/no)? Y # Verify that the device is disconnected.For example:# cfgadm -a ib::1730000007F51,8001,ipib Ap_Id Type Receptacle Occupant Condition ib::1730000007F51,8001,ipib IB-VPPA connected unconfigured unknown # Use the following steps if you want to configure an IB Port, HCA_SVC, or a VPPA device on the system.The example below illustrates how to configure a VPPA device, but similar steps can be used to configure Port and HCA_SVC devices as well. Become superuser. Configure the IB VPPA device.For example:# cfgadm -c configure ib::1730000007F51,8001,ipib Verify that the device is connected.For example:# cfgadm -a ib::1730000007F51,8001,ipib Ap_Id Type Receptacle Occupant Condition ib::1730000007F51,8001,ipib IB-VPPA connected configured okA cfgadm based configure or unconfigure operation of IB Port and HCA_SVC devices is similar to the preceding examples for an IB VPPA device. Use the following steps if you want to remove an IB pseudo device from the system. Become superuser. Unconfigure the IB pseudo device.For example:# cfgadm -c unconfigure ib::ibgen,0 Unconfigure the device: /devices/ib:fabric::ibgen,0 This operation will suspend activity on the IB device Continue (yes/no)? Y # Verify that the device is disconnected.# cfgadm -a ib::ibgen,0 Ap_Id Type Receptacle Occupant Condition ib::ibgen,0 IB-PSEUDO connected unconfigured unknown Use the following steps to configure an IB pseudo device. Become superuser. Configure the IB pseudo device.For example:# cfgadm -yc configure ib::ibgen,0 Verify that the device is connected.For example:# cfgadm -a ib::ibgen,0 Ap_Id Type Receptacle Occupant Condition ib::ibgen,0 IB-PSEUDO connected configured ok The following IB cfgadm plugin command can be invoked to list kernel IB clients using this HCA. Note that the last column would show a “yes” if a kernel IB client uses another HCA. IB Managers and kernel clients that do not use the HCA are shown with an Ap_Id of “-”. Display kernel IB clients of an HCA.For example:$ cfgadm -x list_clients hca:173000007F50 Ap_Id IB Client Alternate HCA ib::1730000007F51D0 ibgen no ib::1730000007F51D1 ibgen no ib::1730000007F51,8001,ipib ibd no ib::ibgen,0 ibgen no - ibdm no - ibmf no - nfs/ib no $ An actual DR of an HCA is beyond the scope of the IB cfgadm plugin. Although DR of an HCA can be achieved by using the plugin of the underlying bus. For example, a PCI based HCA can use the cfgadm_pci command. For more information, see cfgadm_pci1M.However, the IB cfgadm plugin assists in the HCA DR by listing its kernel IB clients as illustrated in steps below. Become superuser. List the kernel IB clients of the HCA.For example:# cfgadm -x list_clients hca:173000007F50 Ap_Id IB Client Alternate HCA ib::1730000007F51D0 ibgen no ib::1730000007F51D1 ibgen no ib::1730000007F51,8001,ipib ibd no ib::ibgen,0 ibgen no - ibdm no - ibmf no - nfs/ib no Unconfigure kernel IB clients, such as Port, VPPA, HCA_SVC, or IOC devices, that do not have alternate HCAs present.For example:# cfgadm -x unconfig_clients hca:1730000008070 Unconfigure Clients of HCA /devices/ib:1730000008070 This operation will unconfigure IB clients of this HCA Continue (yes/no)? y Verify that the kernel IB clients of the HCA are unconfigured.# cfgadm -x list_clients hca:173000007F50 Ap_Id IB Client Alternate HCA - ibdm no - ibmf no - nfs/ib no # Invoke the bus-specific cfgadm plugin to configure the HCA. The exact details are beyond the scope of this chapter. If the p_key table information of an HCA ports changes, for example, additional p_keys are enabled or disabled, InfiniBand Transport Framework (IBTF) and IBDM need to be notified so that their internal p_key databases are updated. The cfgadm command helps update the p_key databases of IBTF and IBDM. For more information, see ibtl7D and ibdm7D. Become superuser. Update the p_key tables.For example:# cfgadm -x update_pkey_tbls -y ib Use the following steps to display the communication services that are currently in use by the IBTF. Become superuser. Display IB communication services.For example:# cfgadm -x list_services ib Port communication services: srp VPPA communication services: ibd HCA_SVC communication services: hnfs Use the following steps to add a new VPPA communication service.Similar steps can be used to add a new HCA_SVC or a port communication service. Become superuser. Add a new VPPA communication service.For example:# cfgadm -o comm=vppa,service=new -x add_service ib Verify that the new service has been added.For example:# cfgadm -x list_services ib Port communication services: srp VPPA communication services: ibd new HCA_SVC communication services: nfs_service # Use the following steps to delete an existing IB Port, HCA_SVC, or a VPPA communication service. Become superuser. Remove a VPPA communication service.For example:# cfgadm -o comm=vppa,service=new -x delete_service ib Verify that the communication service has been removed.For example:# cfgadm -x list_services ib Port communication services: srp VPPA communication services: ibd HCA_SVC communication services: hnfs # Use the following steps to update properties of all the IOC device nodes or for a particular IOC Ap_Id. The properties that can get updated are as follows:port-list port-entries service-id service-name For more information on these properties, see ib7D.Note that these properties may not get updated if there is no configuration change. The following example describes how to update a particular IOC's configuration. If you need to update the configuration of all the IOCs, then specify the static ib Ap_Id instead of the particular IOC Ap_Id. Become superuser. Update the configuration of an IOC.For example:# cfgadm -x update_ioc_conf ib::1730000007F5198 This operation can update properties of IOC devices. Continue (yes/no)? y # Verify that the properties have been updated by running prtconf -v. User Direct Access Programming Library (uDAPL) is a standard API that promotes data center application data messaging performance, scalability, and reliability over Remote Direct Memory Access (RDMA) capable interconnects such as InfiniBand. The uDAPL interface is defined by the DAT collaborative. For more information about the DAT collaborative, go to the following site:http://www.datcollaborative.orgThe Solaris release provides the following uDAPL features:A standard DAT registry library, libdat. For more information, see libdat3LIB. A standard service provider registration file, dat.conf. For more information, see dat.conf4. Support for multiple service providers so that each provider specifies their own uDAPL library path, version number, and so on, in their own service_provider.conf file. For more information, see, service_provider.conf4. An administrative tool, the datadm command, to configure dat.conf. For more information, see datadm1M. A new resource control property, project.max-device-locked-memory, to regulate the amount of locked down physical memory. A naming scheme that uses either IPv4 or IPv6 addresses that leverage the IP infrastructure, such as ARP in IPv4 and neighbor discovery in IPv6, for address resolution. The Solaris uDAPL Interface Adapter directly maps to an IPoIB device instance. Support for the standard Address Translation Scheme that is used by the DAT collaborative community. A uDAPL service provider library to support the Mellanox Tavor Host Channel Adapter with automatic registration to the dat.conf registration file. Supports both SPARC platform and x86 platforms. Become superuser. Confirm that the following packages are installed. Or, install them, if needed.SUNWib – Sun InfiniBand Framework SUNWtavor – Sun Tavor HCA Driver SUNWipoib – Sun IP over InfiniBand SUNWudaplr – Direct Access Transport (DAT) registry package (root) SUNWudaplu – Direct Access Transport (DAT) registry packages (usr) SUNWudapltr – Service Provider for Tavor packages (root) SUNWudapltu – Service Provider for Tavor packages (usr) Select one of the following to plumb the IPoIB interfaces.Manually plumb the interfaces with the ifconfig and datadm commands.For example:# ifconfig ibd1 plumb # ifconfig ibd1 192.168.0.1/24 up # datadm -a /usr/share/dat/SUNWudaplt.conf Automatically plumb the interfaces by doing the following:Create the following file with the appropriate IP address./etc/hostname.ibd1 Reboot the system. You can use the datadm command to maintain the DAT static registry, the dat.conf file. For more information about this file, see dat.conf4.The datadm command can also be used to register or unregister a service provider to the dat.conf file. For more information, see datadm1M.When IPoIB interface adapters are added or removed, run the datadm command to update the dat.conf file to reflect the current state of the system. A new set of interface adapters for all the service providers that are currently installed will be regenerated.Become superuser. Update the DAT static registry after you add or remove IPoIP interface adapters from the system.# datadm -u Display the updated DAT static registry.# datadm Become superuser. Update the DAT static registry after you add Sun's service provider for the Mellanox Tavor Host Channel Adapter.# datadm -a /usr/share/dat/SUNWudaplt.conf Display the updated DAT static registry.# datadm -v Become superuser. Update the DAT static registry after you remove Sun's service provider for the Mellanox Tavor Host Channel Adapter from the system.# datadm -r /usr/share/dat/SUNWudaplt.conf Display the updated DAT static registry.# datadm -v