


STORM                                                   A. Kanevsky, Ed.
Internet-Draft                                                 Dell Inc.
Updates: 5043, 5044 (if approved)                        C. Bestler, Ed.
Intended status: Standards Track                         Nexenta Systems
Expires: June 17, 2012                                          R. Sharp
                                                                   Intel
                                                                 S. Wise
                                                     Open Grid Computing
                                                       December 15, 2011


                 Enhanced RDMA Connection Establishment
                  draft-ietf-storm-mpa-peer-connect-09

Abstract

   This document updates RFC 5043 and RFC 5044 by extending Marker
   Protocol Data Unit (PDU) Aligned Framing (MPA) negotiation for Remote
   Direct Memory Access (RDMA) connection establishment.  The first
   enhancement extends RFC 5044, enabling peer-to-peer connection
   establishment over MPA/ Transmission Control Protocol (TCP).  The
   second enhancement extends both RFC 5043 and RFC 5044, by providing
   an option for standardized exchange of RDMA-layer connection
   configuration.

Status of this Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts is at http://datatracker.ietf.org/drafts/current/.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   This Internet-Draft will expire on June 17, 2012.

Copyright Notice

   Copyright (c) 2011 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal



Kanevsky, et al.          Expires June 17, 2012                 [Page 1]

Internet-Draft   Enhanced RDMA Connection Establishment    December 2011


   Provisions Relating to IETF Documents
   (http://trustee.ietf.org/license-info) in effect on the date of
   publication of this document.  Please review these documents
   carefully, as they describe your rights and restrictions with respect
   to this document.  Code Components extracted from this document must
   include Simplified BSD License text as described in Section 4.e of
   the Trust Legal Provisions and are provided without warranty as
   described in the Simplified BSD License.


Table of Contents

   1.  Introduction . . . . . . . . . . . . . . . . . . . . . . . . .  3
     1.1.  Summary of changes affecting RFC 5044  . . . . . . . . . .  4
     1.2.  Summary of changes affecting RFC 5043  . . . . . . . . . .  4
   2.  Requirements Language  . . . . . . . . . . . . . . . . . . . .  4
   3.  Definitions  . . . . . . . . . . . . . . . . . . . . . . . . .  4
   4.  Motivations  . . . . . . . . . . . . . . . . . . . . . . . . .  7
     4.1.  Standardization of RDMA Read Parameter Configuration . . .  7
     4.2.  Enabling MPA Mode  . . . . . . . . . . . . . . . . . . . .  9
     4.3.  Lack of Explicit RTR in MPA Request/Reply Exchange . . . .  9
     4.4.  Limitations on ULP Workaround  . . . . . . . . . . . . . . 10
       4.4.1.  Transport Neutral APIs . . . . . . . . . . . . . . . . 11
       4.4.2.  Work/Completion Queue Accounting . . . . . . . . . . . 11
       4.4.3.  Host-based Implementation of MPA Fencing . . . . . . . 12
   5.  Enhanced MPA Connection Establishment  . . . . . . . . . . . . 12
   6.  Enhanced MPA Request/Reply Frames  . . . . . . . . . . . . . . 13
   7.  Enhanced SCTP Session Control Chunks . . . . . . . . . . . . . 14
   8.  MPA Error Reporting  . . . . . . . . . . . . . . . . . . . . . 16
   9.  Enhanced RDMA Connection Establishment Data  . . . . . . . . . 16
     9.1.  IRD and ORD Negotiation  . . . . . . . . . . . . . . . . . 17
     9.2.  Peer-to-Peer Connection Negotiation  . . . . . . . . . . . 19
     9.3.  Enhanced Connection Negotiation Flow . . . . . . . . . . . 20
   10. Interoperability . . . . . . . . . . . . . . . . . . . . . . . 21
   11. IANA Considerations  . . . . . . . . . . . . . . . . . . . . . 22
   12. Security Considerations  . . . . . . . . . . . . . . . . . . . 22
   13. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 22
   14. References . . . . . . . . . . . . . . . . . . . . . . . . . . 22
     14.1. Normative References . . . . . . . . . . . . . . . . . . . 22
     14.2. Informative References . . . . . . . . . . . . . . . . . . 23
   Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 24










Kanevsky, et al.          Expires June 17, 2012                 [Page 2]

Internet-Draft   Enhanced RDMA Connection Establishment    December 2011


1.  Introduction

   When used over Transmission Control Protocol (TCP), the current
   Remote Direct Data Placement (RDDP) [RFC5041] suite of protocols
   relies on MPA [RFC5044] protocol for both connection establishment
   and for markers for TCP layering.

   A typical model for establishing an RDMA connection has the following
   steps:

   o  The passive side (responder) Upper Layer Protocol (ULP) listens
      for connection requests.

   o  The active side (initiator) ULP submits a connection request using
      an RDMA endpoint, the desired destination and the parameters to be
      used for the connection.  Those parameters include both RDMA layer
      characteristics, such as the number of simultaneous RDMA Read
      Requests to be allowed and application specific data.

   o  The passive side ULP receives a connection request, which includes
      the identity of the active side and the requested connection
      characteristics.  The passive side ULP uses this information to
      decide whether to accept the connection, and if it is to be
      accepted, how to create and/or configure the local RDMA endpoint.

   o  If accepting, responder submits its acceptance of the connection
      request, which in turn, generates the accept message to initiator.
      This responder accept operation includes the RDMA endpoint to be
      used and the connection characteristics (both the RDMA
      configuration and any application specific private data to be
      transferred to initiator).

   o  The active side receives confirmation that the connection has been
      accepted, what the configured connection characteristics are, and
      any application supplied private data.

   Currently, MPA only supports a client-server model for connection
   establishment, forcing peer-to-peer applications to interact as
   though they had a client/server relationship.  In addition
   negotiation of some of Remote Direct Memory Access Protocol (RDMAP)
   [RFC5040] specific parameters are left to ULP negotiation.  Providing
   an optional ULP-independent format for exchanging these parameters
   would be of benefit to transport neutral Remote Direct Memory Access
   (RDMA) applications.







Kanevsky, et al.          Expires June 17, 2012                 [Page 3]

Internet-Draft   Enhanced RDMA Connection Establishment    December 2011


1.1.  Summary of changes affecting RFC 5044

   This draft enhances [RFC5044] MPA connection setup protocol.  First,
   it adds exchange and negotiation of the parameters necessary to
   support RDMA Read Requests.  Second, it adds a message that serves as
   a Ready to Receive (RTR) indication from the initiator to the
   responder as the last message of connection establishment and adds
   negotiation of an which type of message to use to carry the RTR
   indication into MPA request/reply frames.

1.2.  Summary of changes affecting RFC 5043

   This draft enhances [RFC5043] by adding new Enhanced Session Control
   Chunks that extends the currently defined Chunks with the addition of
   Inbound RDMA Read Queue Depth (IRD) and Outbound RDMA Read Queue
   Depth (ORD) negotiation.


2.  Requirements Language

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
   document are to be interpreted as described in [RFC2119].


3.  Definitions

   Active Side:  See Initiator.

   Consumer:  The ULPs or applications that lie above MPA and Direct
      Data Placement (DDP).  The Consumer is responsible for making TCP
      or SCTP connections, starting MPA and DDP connections, and
      generally controlling operations.  See [RFC5044] and [RFC5043].

   CRC:  Cyclic Redundancy Check

   Completion Queue (CQ):  A consumer accessible queue where the RDMA
      device reports completions of Work Requests.  A Consumer is able
      to reap completions from a CQ without requiring per transaction
      support from the kernel or other privileged entity.  See [RDMAC].

   Completion Queue Entry (CQE):  Transport and device specific
      representation of a Work Completion.  A Completion Queue holds
      CQEs.  See [RDMAC].







Kanevsky, et al.          Expires June 17, 2012                 [Page 4]

Internet-Draft   Enhanced RDMA Connection Establishment    December 2011


   FULPDU:  Framed Upper Layer Protocol PDU.  See FPDU of [RFC5044].

   Inbound RDMA Read Request Queue (IRRQ):  A queue that is associated
      with an RDMA Connection that tracks active incoming simultaneous
      RDMA Read Request Messages.  See [RDMAC].

   Inbound RDMA Read Queue Depth (IRD):  The maximum number of incoming
      simultaneous RDMA Read Request Messages an RDMA connection can
      handle.  See [RDMAC].

   Initiator:  The endpoint of a connection that sends the MPA Request
      Frame.  Initiator is the active side of the connection
      establishment.  See [RFC5044].

   IRD:  See Inbound RDMA Read Queue Depth.

   MPA Fencing:  MPA responder Connection Establishment logic that
      ensures that no ULP messages will be transferred until the
      initiator first message has been received.

   MPA Request Frame:  Data sent from the MPA initiator to the MPA
      responder during the Startup Phase.  See [RFC5044].

   MPA Reply Frame:  Data sent from the MPA responder to the MPA
      initiator during the Startup Phase.  See [RFC5044].

   ORD:  See Outbound RDMA Read Queue Depth.

   Outbound RDMA Read Queue Depth (ORD):  The maximum number of
      simultaneous RDMA Read Requests that can be issued for the RDMA
      connection.  This should be less than or equal to the peer's IRD.
      See [RDMAC].

   Passive Side:  See Responder.

   Private Data:  A block of data exchanged between MPA endpoints during
      initial connection setup.  See [RFC5044].

   Queue Pair (QP):  The traditional name for a local Endpoint in a
      [VIA] derived local interface.  A Queue Pair is the set of Work
      Queues associated exclusively with a single Endpoint.  The Send
      Queue (SQ), Receive Queue (RQ) and Inbound RDMA Read Queue (IRQ)
      are considered to be part of the Queue Pair.  The potentially
      shared Completion Queue (CQ) and Shared Receive Queue (SRQ) are
      not.  See [RDMAC].






Kanevsky, et al.          Expires June 17, 2012                 [Page 5]

Internet-Draft   Enhanced RDMA Connection Establishment    December 2011


   Remote Peer:  The MPA protocol implementation on the opposite end of
      the connection.  Used to refer to the remote entity when
      describing protocol exchanges or other interactions between two
      Nodes.  See [RFC5044].

   Responder:  The connection endpoint that responds to an incoming MPA
      connection request (the MPA Request Frame).  Responder is the
      passive side of the connection establishment.  See [RFC5044].

   Ready to Receive (RTR):  RTR is an indication provided by the last
      connection establishment message sent from the initiator to the
      responder.  An RTR indicates that the initiator is ready to
      receive messages and that connection establishment is completed.

   Startup Phase:  The initial exchanges of an MPA connection that
      serves to more fully identify MPA endpoints to each other and pass
      connection specific setup information to each other.  See
      [RFC5044].

   Shared Receive Queue(SRQ):  A shared pool of Receive Work Requests
      posted by the Consumer that can be allocated by multiple RDMA
      endpoints (Queue Pair).  See [RDMAC].

   Tagged (DDP) Message:  - A DDP Message that targets a Tagged Buffer
      that is explicitly Advertised to the Remote Peer through exchange
      of an STag (memory handle), offset in the memory region identified
      by STag, and length [RFC5040].

   Untagged (DDP) Message:  - A DDP Message that targets an Untagged
      Buffer associated with a queue specified by Queue Number (QN).
      [RFC5040].

   Work Queue:  An element of a [VIA] derived local interface that
      allows user-space applications to submit Work Requests directly to
      network hardware.  Specific Work Queues include the Send Queue
      (SQ) for transmit requests, Receive Queue (RQ) for receive
      requests specific to a single Endpoint and Shared Receive Queues
      (SRQs) for receive requests that can be allocated by one or more
      Endpoints.  See [RDMAC].

   Work Queue Element (WQE):  Transport and device specific
      representation of a Work Request.  See [RDMAC].

   Work Request:  An elementary object used by Consumers to enqueue a
      requested operation (WQEs) onto a Work Queue.  See [RDMAC].






Kanevsky, et al.          Expires June 17, 2012                 [Page 6]

Internet-Draft   Enhanced RDMA Connection Establishment    December 2011


4.  Motivations

   The goal of this draft is twofold.  One is to extend support from the
   current client-server model for RDMA connection setup to a peer-to-
   peer model.  The second is to add negotiation of RDMA Read queue size
   for both sides of an RDMA connection.

4.1.  Standardization of RDMA Read Parameter Configuration

   Most RDMA applications are developed using a transport neutral
   Application Programming Interface (API) to access RDMA services based
   on a "queue pair" paradigm as originally defined by the Virtual
   Interface Architecture [VIA], refined by the Direct Access
   Programming Library [DAPL] and most commonly deployed with the
   OpenFabrics API [OFA].

   These transport neutral APIs seek to provide a common set of RDMA
   services whether the underlying transport is, for example, RDDP over
   MPA, RDDP over SCTP or InfiniBand.

   The common model for establishing an RDMA connection has the
   following steps:

   o  The passive side ULP listens for connection requests.

   o  The active side ULP submits a connection request using an RDMA
      endpoint ("queue pair"), the desired destination and the
      parameters to be used for the connection.  Those parameters
      include both RDMA layer characteristics, such as the RDMA Read
      credits to be allowed and application specific data (typically
      referred to as "private data").

   o  The passive side ULP receives a connection request, which includes
      the identity of the active side and the requested connection
      characteristics.  The passive side ULP uses this information to
      decide whether to accept the connection, and if it is to be
      accepted, how to create and/or configure the RDMA endpoint.

   o  If accepting, the passive side ULP submits its acceptance of the
      connection request.  This local accept operation includes the RDMA
      endpoint to be used and the connection characteristics (both the
      RDMA configuration and any application specific private data to be
      returned).

   o  The active side receives confirmation that the connection has been
      accepted, what the configured connection characteristics are, and
      any application supplied private data.




Kanevsky, et al.          Expires June 17, 2012                 [Page 7]

Internet-Draft   Enhanced RDMA Connection Establishment    December 2011


   As currently defined, DDP connection establishment requires the ULP
   to encode the RDMA configuration in the application specific private
   data.  This results in undesirable duplication of logic to cover both
   InfiniBand and RDDP, and to specify the extraction of the RDMA
   characteristics from the ULP for each specific Upper Layer Protocol.

   Both RDDP and InfiniBand support an initial private data exchange,
   therefore a standard definition of the RDMA characteristics within
   the private data section would enable common connection establishment
   APIs to format the RDMA characteristics based on the same API
   information used when establishing either protocol to form the
   connection.  The application would then only have to indicate that it
   was using this standard format to enable common connection
   establishment procedures to apply common code to properly parse these
   fields and configure the RDMA endpoints accordingly.  Exchange of
   parameters necessary to perform RDMA Read operations is a common
   usage of the initial private data exchange.

   One of the RDMA operations that is defined in [RDMAC] is an RDMA
   Read.  RDMA Read operations are performed using an untagged message
   sent from a Queue Pair (QP) on the local endpoint to a QP on the
   remote endpoint targeting the Inbound RDMA Read Request Queue (QN=1
   or Inbound RDMA Read Request Queue (IRRQ)) associated with the
   connection.  RDMA Read responses transfer data associated with each
   RDMA Read Request from the remote endpoint to the local endpoint
   using tagged messages.  An inbound RDMA Read Request remains on the
   IRRQ from the time that it is received until the time that the last
   tagged message associated with the RDMA request is acknowledged.  The
   IRRQ is associated with a QP but is not a Work Queue.  Instead the
   IRRQ is a standalone queue that is used to manage RDMA read requests
   associated with a QP.  See [RDMAC] section 6 for more information
   regarding QPs and IRRQ.  One of the characteristics that must be
   configured for a QP is the size of the IRRQ.  This parameter is
   called the Inbound RDMA Read Queue Depth (IRD).  Another
   characteristic of a QP that must be configured a local limit on the
   number of simultaneous outbound RDMA Read Requests based on the size
   of the remote endpoint QP's IRRQ.  This parameter is call the
   Outbound RDMA Read Queue Depth (ORD).  ORD is used to limit the
   number of simultaneous RDMA read requests such that the local
   endpoint does not overrun the remote endpoint's IRRQ depth or IRD.
   Note that outbound RDMA Reads are submitted to a QP's Send Queue at
   the local peer, not to a separate outbound RDMA read request queue on
   the local peer.  The local endpoint uses ORD to strictly limit
   simultaneous read requests so that IRRQ overruns do not occur at the
   remote endpoint.

   Determination of the values of the ORD and IRD are left to the ULP by
   the current RDDP suite of protocols and also by [RDMAC].  Since this



Kanevsky, et al.          Expires June 17, 2012                 [Page 8]

Internet-Draft   Enhanced RDMA Connection Establishment    December 2011


   negotiation of ORD and IRD is typical, it is desirable to provide a
   common mechanism described in this draft.

4.2.  Enabling MPA Mode

   MPA defines encoding of DDP Segments in Framed Upper Layer Protocol
   PDUs (FULPDUs).  Generation of FULPDUs requires the ability to
   periodically insert MPA Markers and to generate the MPA CRC-32c for
   each frame.  Reception may require parsing/removing the markers after
   using them to identify MPA Frame boundaries, and validation of the
   MPA-CRC32c.

   A major design objective for MPA was to ensure that the resulting TCP
   stream would be a fully compliant TCP stream for any and all TCP-
   aware middle-boxes.  The challenge is that while only some TCP
   payload streams are a valid stream of MPA FULPDUs, any sequence of
   bytes is a valid TCP payload stream.  The determination that a given
   stream is in a specific MPA mode cannot be made at the MPA or TCP
   layer.  Therefore enabling of MPA mode is handled by the ULP.

   The MPA protocol can be viewed as having two parts.

   o  a specification of generation and reception of MPA FULPDUs.  This
      is unchanged by enhanced RDMA connection establishment.

   o  a pre-MPA exchange of messages to enable a specific MPA mode for
      the TCP connection.  Enhanced RDMA connection establishment
      extends this protocol with two new features.

   In typical implementations, generation and reception of MPA FULPDUs
   is handled by hardware.  The exchange of the MPA Request and Reply
   frames is then handled by host software.  As will be explained, this
   implementation split impedes applications which are not compatible
   with the client-server assumptions in the current MPA Request/Reply
   exchange.

4.3.  Lack of Explicit RTR in MPA Request/Reply Exchange

   The exchange of MPA Request and Reply messages to place a TCP
   connection in MPA mode is specified in [RFC5044].  This protocol
   provides many benefits to the design of MPA FULPDU hardware:

   o  The ULP is responsible for specifying the exact MPA Mode (Markers
      enabled or disabled, CRC-32c enabled or suppressed) and the point
      in the TCP streams (inbound and outbound) where MPA frames will
      begin.





Kanevsky, et al.          Expires June 17, 2012                 [Page 9]

Internet-Draft   Enhanced RDMA Connection Establishment    December 2011


   o  Before the first MPA frame is transmitted, all pre-MPA mode TCP
      payload will have been acknowledged by the peer.  Therefore it is
      never necessary to generate a retransmission that mixes pre-MPA
      and MPA payload.

   o  Before MPA reception is enabled, all incoming pre-MPA mode TCP
      payload will have been acknowledged.  Therefore the host will
      never receive a TCP segment that mixes pre-MPA and MPA payload.

   The limitation of the current MPA Request/Reply exchange is that it
   does not define a Ready to Receive (RTR) indication that the active
   side would send, so that the passive side can know that the last non-
   MPA payload (the MPA Reply) had been received.

   Instead, the role of an RTR indication is piggy-backed on the first
   MPA FULPDU sent by the active side.  This is actually a valuable
   optimization for all applications that fit the classic client/server
   model.  The client only initiates the connection when it has a
   request to send to the server, and the server has nothing to send
   until it has received and processed the client request.

   Even applications where the server sends some configuration data
   immediately can easily send the same information as application
   private data in the MPA Reply.  So the currently defined exchange
   works for almost all applications.

   Many peer-to-peer applications, especially those involving cluster
   calculations (frequently using Message Passing Interface (MPI)
   [UsingMPI], or [RDS]), have no natural client or server roles
   ([PPMPI], [OpenMP]).  Typically one member of the cluster is
   arbitrarily selected to initiate the connection when the distributed
   task is launched, while the other accepts it.  At startup time,
   however, there is no way to predict which node will have the first
   message to actually send.  Establishing the connections immediately,
   however, is valuable because it reduces latency once results are
   ready to transmit and it validates connectivity throughout the
   cluster.

   The lack of an explicit RTR indication in the MPA Request/Reply
   exchange forces all applications to have a first message from the
   connection initiator, whether this matches the application
   communication model or not.

4.4.  Limitations on ULP Workaround

   The requirement that the RDMA connection initiator sends the first
   message does not appear to be onerous on first examination.  The
   natural question is why the application layer would not simply



Kanevsky, et al.          Expires June 17, 2012                [Page 10]

Internet-Draft   Enhanced RDMA Connection Establishment    December 2011


   generate a dummy message when there was no other message to submit.

   There are three factors that make this workaround unsuitable for many
   peer-to-peer applications.

   o  Transport Neutral APIs.

   o  Work/Completion Queue Accounting.

   o  Host-based implementation of MPA Fencing.

4.4.1.  Transport Neutral APIs

   Many of these applications access RDMA services using a transport
   neutral API such as [DAPL] or [OFA].  Only RDDP over TCP [RFC5044]
   has a first message requirement.  Other RDMA transports, including
   RDDP over SCTP (see [RFC5043]) and InfiniBand (see [IBTA]), do not.

   Application or middleware communications can be expressed as
   transport neutral RDMA operations, allowing lower software layers to
   translate to transport and device specifics.  Having a distinct extra
   message that is required only for one transport undermines the
   application's goal of being transport neutral.

4.4.2.  Work/Completion Queue Accounting

   RDMA local APIs conventionally use work queues to submit requests
   (work queue elements or WQEs) and to asynchronously receive
   completions (in completion queues or CQs).

   Each work request can generate a completion queue entry (CQE).
   Completions for successful transmit work requests are frequently
   suppressed, but the completion queue capacity must account for the
   possibility that each will complete in error.  A completion queue can
   receive completions from multiple work queues.

   Completion Queues are defined so as to allow hardware RDMA
   implementations to generate CQEs directly to a user-space mapped
   buffer.  This enables a user-space RDMA consumer to reap completions
   without requiring kernel intervention.

   A hardware RDMA implementation cannot reasonably wait for an
   available slot in the completion queue.  The queue must be sized such
   that an overflow will not occur.  When an overflow does occur it is
   considered catastrophic and will typically require tearing down all
   RDMA connections using that CQ.

   This style of interface is very efficient, but places a burden on the



Kanevsky, et al.          Expires June 17, 2012                [Page 11]

Internet-Draft   Enhanced RDMA Connection Establishment    December 2011


   application to properly size each Completion Queue to match the Work
   Queues that feed it.

   While the format of both WQEs and CQEs is transport and device
   dependent, a transport neutral API can deal with WQEs and CQEs as
   abstract transport and device neutral objects.  Therefore the number
   of WQEs and CQEs required for an application can be transport and
   device neutral.

   The capacity of the work queues and completion queues can be
   calculated in an abstract transport/device neutral fashion.  If a
   dummy operation approach was used, it would require lower layers to
   know the usage model, and would disrupt the calculations by inserting
   a dummy "operation" Work Request and filtering out the matching
   completion.  The lower layer does not know the usage model on which
   the queue sizes are built, nor does it know how frequently an
   insertion will be required.

4.4.3.  Host-based Implementation of MPA Fencing

   Many hardware implementations of RDDP using MPA/TCP do not handle the
   MPA Request/Reply exchange in hardware, rather they are handled by
   the host processor in software.  With such designs it is common for
   the MPA Fencing to be implemented in the user-space device-specific
   library (commonly referred to as a 'User Verbs' library or module).

   When the generation and reception of MPA FULPDUs is already dedicated
   to hardware, a Work Completion can only be generated by an untagged
   message since arrival of a message for tagged buffer does not
   necessarily generate a completion and is done without any interaction
   with ULP [RFC5040].


5.  Enhanced MPA Connection Establishment

   Below we provide an overview of Enhanced Connection Setup.  The goal
   is to allow standard negotiation of ORD/IRD setting on both sides of
   the RDMA connection and/or to negotiate the initial data transfer
   operation by the initiator when the existing 'client sends first'
   rule does not match application requirements.

   The RDMA connection initiator sends an MPA Request, as specified in
   [RFC5044]; the new format defined here allows for:

   o  Standardized negotiation of ORD and IRD.

   o  Negotiation of RTR functionality and the RDMA message type to use
      as the RTR indication.



Kanevsky, et al.          Expires June 17, 2012                [Page 12]

Internet-Draft   Enhanced RDMA Connection Establishment    December 2011


   The RDMA connection responder processes the MPA Request and generates
   an MPA Reply, as specified in [RFC5044]; the new format completes the
   negotiation.

   The local interface needs to provide a way for a ULP to request the
   use of explicit RTR indication per-application or per-connection
   basis when an explicit RTR indication will be required.  Piggy-
   backing the RTR on a Client's first message is a valuable
   optimization for most connections.

   The RDMA connection initiator MUST NOT allow any later FULPDUs to be
   transmitted before the RTR indication.  One method to achieve that is
   to delay notifying the ULP that the RDMA connection has been
   established until after any required RTR indication has been
   transmitted.

   All MPA exchanges are performed via TCP prior to RDMA establishment,
   and are therefore signaled via TCP and not via RDMA completion.


6.  Enhanced MPA Request/Reply Frames

   Enhanced RDMA connection establishment uses an alternate format for
   MPA Requests and Replies, as follows:

       0                   1                   2                   3
        0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    0  |                                                               |
       +         Key (16 bytes containing "MPA ID Req Frame")          +
    4  |      (4D 50 41 20 49 44 20 52 65 71 20 46 72 61 6D 65)        |
       +         Or  (16 bytes containing "MPA ID Rep Frame")          +
    8  |      (4D 50 41 20 49 44 20 52 65 70 20 46 72 61 6D 65)        |
       +                                                               +
    12 |                                                               |
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    16 |M|C|R|S| Res   |     Rev       |          PD_Length            |
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
       |                                                               |
       ~                                                               ~
       ~                   Private Data                                ~
       |                                                               |
       |                               +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
       |                               |
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+






Kanevsky, et al.          Expires June 17, 2012                [Page 13]

Internet-Draft   Enhanced RDMA Connection Establishment    December 2011


   Key:  Unchanged from [RFC5044].

   M: Unchanged from [RFC5044].

   C: Unchanged from [RFC5044].

   R: Unchanged from [RFC5044].

   S: One if the Private Data begins with the enhanced RDMA connection
      establishment data.  Zero otherwise.

   Res:  One bit smaller than in [RFC5044], otherwise unchanged.  In
      [RFC5044] 'Res' field, in which the newly defined 'S' bit resides,
      is reserved for future use.  [RFC5044] specifies that 'RES' MUST
      be set to zero when sending, and MUST NOT be checked on reception,
      making use of S bit backwards compatible with the original MPA
      frame format.  When the S bit is set to zero, no additional
      private data is used for enhanced RDMA connection establishment,
      and therefore the resulting MPA request and reply frames are
      identical to the unenhanced protocol.

   Rev:  This field contains the revision of MPA.  To use any enhanced
      connection establishment feature this MUST be set to two or
      higher, If no enhanced connection establishment features are
      desired it MAY be set to one.  A host accepting MPA connections
      MUST continue to accept MPA Requests with version one even if it
      supports version two.

   PD_Length:  Unchanged from [RFC5044].  This is the total length of
      the Private Data field, including the enhanced RDMA connection
      establishment data if present.

   Private Data:  Unchanged from [RFC5044].  However, if the 'S' flag is
      set, Private Data MUST begin with enhanced RDMA connection
      establishment data (see Section 9).


7.  Enhanced SCTP Session Control Chunks

   Enhanced RDMA Connection Establishment uses the first 32 bits of the
   Private data field for IRD and ORD negotiation in the "DDP Stream
   Session Initiate" and "DDP Stream Session Accept" SCTP Session
   Control Chunks.

   The type of the SCTP Session Control Chunk is defined by a Function
   Code (see [RFC4960]).  [RFC5043] already defines codes for 'DDP
   Stream Session Initiate' and 'DDP Stream Session Accept', which are
   equivalent to a MPA Request Frame and an accepting MPA Reply Frame.



Kanevsky, et al.          Expires June 17, 2012                [Page 14]

Internet-Draft   Enhanced RDMA Connection Establishment    December 2011


   Enhanced RDMA connection establishment requires three additional
   Function codes listed below:

   Enhanced DDP Stream Session Initiate:  0x005

   Enhanced DDP Stream Session Accept:  0x006

   Enhanced DDP Stream Session Reject:  0x007

   The Enhanced Reject function code MUST be used to indicate rejection
   of enhanced DDP stream session for a configuration that would have
   been accepted for unenhanced DDP Stream Session negotiation.

   The Enhanced DDP stream session establishment follows the same rules
   as the standard DDP stream session establishment as defined in
   [RFC5043].  ULP-supplied Private Data MUST be included for Enhanced
   DDP Stream Session Initiate, Enhanced DDP Stream Session Accept, and
   Enhanced DDP Stream Session Reject messages, and MUST follow the
   enhanced RDMA connection establishment data in the DDP Stream Session
   Initiate and the Enhanced DDP Stream Session Accept messages.

   Private Data length MUST NOT exceed 512 bytes in any message,
   including enhanced RDMA connection establishment data.

   Private Data MUST NOT be included in the DDP Stream Session TERM
   message.

   Received Extended DDP Stream Session Control messages SHOULD be
   reported to the ULP.  If reported, any supplied Private Data MUST be
   available for the ULP to examine.  For example, a received Extended
   DDP Stream Session Control message is not reported to ULP if none of
   the requested RTR indication types are supported by receiver.  In
   this case, Provider MAY generate reject reply message indicating
   which RTR indication types it supports.

   The enhanced DDP stream management MUST use the DDP stream session
   termination function code to terminate a stream established using
   enhanced DDP stream session function codes.

   [RFC5043] already supports either side sending the first DDP Message
   since the Payload Protocol Identifier (PPID) already distinguishes
   between Session Establishment and DDP Segments.  The enhanced RDMA
   Connection Establishment provides to the ULP a transport independent
   way to support peer-to-peer model.

   The following additional Legal Sequences of DDP Stream Session
   messages are defined:




Kanevsky, et al.          Expires June 17, 2012                [Page 15]

Internet-Draft   Enhanced RDMA Connection Establishment    December 2011


   o  Enhanced Active/Passive Session Accepted: as with section 6.2 of
      [RFC5043], but with the extended opcodes as defined in this
      document.

   o  Enhanced Active/Passive Session Rejected: as with section 6.3 of
      [RFC5043], but with the extended opcodes as defined in this
      document.

   o  Enhanced Active/Passive Session Non-ULP Rejected: as with section
      6.4 of [RFC5043], but with the extended opcodes as defined in this
      document.


8.  MPA Error Reporting

   The RDMA connection establishment protocol is layered upon [RFC5040]
   and [RFC5041].  Any enhanced RDMA connection establishment error
   generates an MPA termination message to a peer.  [RFC5040] defines a
   triplet of protocol layers, error types and error codes for error
   specification.  MPA negotiation for RDMA connection establishment
   uses the following layer and error type for MPA error reporting:

   Layer:      0x2 - LLP
   Error Type: 0x0 - MPA

   While [RFC5044] defines four error codes, [RFC5043] does not define
   any.  Enhanced RDMA connection establishment extends [RFC5044] error
   codes by adding three new error codes.  Thus, enhanced RDMA
   connection establishment is backward compatible with both [RFC5043]
   and [RFC5044].

   The following error codes are defined for enhanced RDMA connection
   establishment negotiation:

   Error Code         Description
   --------------------------------------------------------
   0x05               Local catastrophic
   0x06               Insufficient IRD resources
   0x07               No matching RTR option


9.  Enhanced RDMA Connection Establishment Data

   Enhanced RDMA Connection Establishment places the following 32 bits
   at the beginning of the Private data field of the MPA Request and
   Reply Frames or the "DDP Stream Session Initiate" and "DDP Stream
   Session Accept" SCTP Session Control Chunks.  ULP specified private
   data follows this field.  The maximum amount of ULP specified private



Kanevsky, et al.          Expires June 17, 2012                [Page 16]

Internet-Draft   Enhanced RDMA Connection Establishment    December 2011


   data is therefore reduced by 4 bytes.  Note that this field MUST be
   sent in network byte order, with IRD and ORD encoded as 14 bit
   unsigned integers.

       0                   1                   2                   3
        0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    0  |A|B|        IRD                |C|D|        ORD                |
    4  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+



   IRD:  Inbound RDMA Read Queue Depth.

   ORD:  Outbound RDMA Read Queue Depth.

   A: Control Flag for connection model.

   B: Control Flag for use of a zero length FULPDU (Send) RTR
      indication.

   C: Control Flag for use of a zero length RDMA Write RTR indication.

   D: Control Flag for use of a zero length RDMA Read RTR indication.

9.1.  IRD and ORD Negotiation

   IRD and ORD are used for negotiation of Inbound RDMA Read Request
   Queue depths for both endpoints of the RDMA connection.  IRD is used
   to configure the depth of the Inbound RDMA Read Request Queue (IRRQ)
   on each endpoint.  ORD is used to limit the number of simultaneous
   outbound RDMA Read Requests allowed at at given point in time in
   order to avoid IRRQ overruns at the remote endpoint.  In order to
   describe the negotiation of both local endpoint and remote endpoint
   ORD and IRD values, four terms are defined:

   Initiator IRD:  IRD value sent in the MPA request or "DDP Stream
      Session Initiate" SCTP Session Control Chunk.  This is the value
      of the initiator's IRD at the time of the MPA Request generation.
      The responder sets its local ORD value to this value or less.
      Initiator IRD is the maximum number of simultaneous inbound RDMA
      Read Requests which the initiator can support for the requested
      connection.

   Initiator ORD:  ORD value in the MPA request or "DDP Stream Session
      Initiate" SCTP Session Control Chunk.  This is the initial value
      of the initiator's ORD at the time of the MPA Request generation
      and also a request to the responder to support a responder IRD of



Kanevsky, et al.          Expires June 17, 2012                [Page 17]

Internet-Draft   Enhanced RDMA Connection Establishment    December 2011


      at least this value.  Initiator ORD is the maximum number of
      simultaneous outbound RDMA Read operations that the initiator
      desires the responder to support for the requested connection.

   Responder IRD:  IRD value returned in the MPA reply or "DDP Stream
      Session Accept" SCTP Session Control Chunk.  This is the actual
      value that the responder set for its local IRD.  This value is
      greater than or equal to initiator ORD for successful
      negotiations.  Responder IRD is the maximum number of simultaneous
      inbound RDMA Read Requests that the responder actually can support
      for the requested connection.

   Responder ORD:  ORD value returned in the MPA reply or "DDP Stream
      Session Accept" SCTP Session Control Chunk.  This is the actual
      value that the responder used for ORD and is less than or equal to
      initiator IRD for successful negotiations.  Responder ORD is the
      maximum number of simultaneous outbound RDMA Read operations that
      the responder will allow for the requested connection.

   The relationships between these parameters after a successful
   negotiation is complete are the following:

   initiator ORD <= responder IRD

   responder ORD <= initiator IRD

   The responder and initiator MUST pass the peer's provided IRD and ORD
   values to the ULP, in addition to using the values as calculated by
   the preceding rules.

   Responder ORD SHOULD be set to a value less than or equal to
   initiator IRD.  If initiator ORD is insufficient to support the
   selected connection model, responder IRD MAY be increased, for
   example if initiator ORD is 0 (RDMA Reads will not be used by the
   ULP) and the responder supports use of a zero length RDMA Read RTR
   indication, then responder IRD can be set to 1.  The responder MUST
   set its ORD at most to initiator IRD.  The responder MAY reject the
   connection request if initiator IRD is not sufficient for the ULP
   required ORD and specify the required ORD in the MPA Reject frame
   responder ORD.  Thus, the TERM message MUST contain Layer 2, Error
   Type 0, Error Code 6.

   Upon receiving the MPA Accept frame from the responder, the initiator
   MUST set its IRD at least to responder ORD and its ORD at most to
   responder IRD.  If the initiator does not have sufficient resources
   for the required IRD, it MUST send a TERM message to the responder
   indicating insufficient resources, and terminate the connection due
   to insufficient resources.  Thus, the TERM message MUST contain Layer



Kanevsky, et al.          Expires June 17, 2012                [Page 18]

Internet-Draft   Enhanced RDMA Connection Establishment    December 2011


   2, Error Type 0, Error Code 6.

   The initiator MUST pass the responder provided IRD and ORD to the ULP
   for both MPA Accept and Reject messages.  The initiator ULP can
   decide its course of action.  For example, the initiator ULP may
   terminate the established connection and renegotiate responder ORD.

   An all ones value (0x3FFF) indicates that automatic negotiation of
   the IRD or ORD is not desired, and that the ULP will be responsible
   for it.  The responder MUST respond to an initiator ORD value of
   0x3FFF by leaving its local endpoint IRD value unchanged, and setting
   IRD to 0x3FFF in its reply message.  The initiator MUST leave its
   local endpoint ORD value unchanged upon receiving a responder IRD
   value of 0x3FFF.  The responder MUST respond to an initiator IRD
   value of 0x3FFF by leaving its local endpoint ORD value unchanged,
   and setting ORD to 0x3FFF in its reply message.  The initiator MUST
   leave its local endpoint IRD value unchanged upon receiving a
   responder ORD value of 0x3FFF.

9.2.  Peer-to-Peer Connection Negotiation

   Control Flag A value 1 indicates that a peer-to-peer connection model
   is being performed, and value 0 indicates a client-server model.
   Control Flag B value 1 indicates that a zero length FULPDU (Send) RTR
   indication is requested for the initiator and supported by the
   responder, respectively, 0 otherwise.  Control Flag C value 1
   indicates that a zero length RDMA Write RTR indication is requested
   for the initiator and supported by the responder, respectively, 0
   otherwise.  Control Flag D value 1 indicates that a zero length RDMA
   Read RTR indication is requested for the initiator and supported by
   the responder, respectively, 0 otherwise.  The initiator MUST set
   Control Flag A to 1 for peer-to-peer model.  The initiator MUST set
   each Control Flag B, C and D to 1 for each of the options it
   supports, if Control Flag A is set to 1.

   The responder MUST support at least one RTR indication option if it
   supports Enhanced RDMA connection establishment.  If Control Flag A
   is 1 in the MPA request message then the responder MUST set Control
   Flag A to 1 in the MPA reply message.  For each initiator supported
   RTR indication option the responder SHOULD set the corresponding
   Control Flag if the responder can support that option in an MPA
   reply.  The responder is not required to specify all RTR indication
   options it supports.  The responder MUST set at least one RTR
   indication option if it supports more than one initiator specified
   RTR indication option.  The responder MAY include additional RTR
   indication options it supports, even if not requested by any
   initiator specified RTR indication options.  If the responder does
   not support any of the initiator specified RTR indication options



Kanevsky, et al.          Expires June 17, 2012                [Page 19]

Internet-Draft   Enhanced RDMA Connection Establishment    December 2011


   then the responder MUST set at least one RTR indication type option
   it supports.

   Upon receiving the MPA accept frame with Control Flag A set to 1, the
   initiator MUST generate one of the negotiated RTR indications.  If
   the initiator is not able to generate any of the responder supported
   RTR indications, then it MUST send a TERM message to the responder
   indicating failure to negotiate a mutually compatible connection
   model or RTR option, and terminate the connection.  Thus, the TERM
   message MUST contain Layer 2, Error Type 0, Error Code 7.  The ULP
   can negotiate a ULP level RTR indication when a Provider level RTR
   indication cannot be negotiated.

   The initiator MUST set Control Flag A to 0 for client-server model.
   The responder MUST set Control Flag A to 0 if Control Flag A is 0 in
   request.  If Control Flag A is set to 0 then Control Flags B, C and D
   MUST also be set to 0.  On reception if Control Flag A is set to 0
   then Control Flags B, C, and D MUST be ignored.

9.3.  Enhanced Connection Negotiation Flow

   The RTR indication type and ORD/IRD negotiation follows the following
   order:

   initiator (MPA Request) -->  Set Control Flag A to 1 to indicate
      peer-to-peer connection model and initiator IRD, ORD setting on
      local Endpoint of the connection.  Set Control Flags B, C, and D
      to 1 for each initiator-supported option of RTR indication.

   responder (MPA Reply) <--  Match the initiator Control Flag A value
      and set ORD/IRD to the responder local endpoint values based upon
      the initiator initial ORD/IRD values and the number of
      simultaneous RDMA Read Requests required by the ULP.  Sets Control
      Flags B, C, and D to 1 for responder-supported options of RTR
      indication options for peer-to-peer connection model and sets the
      responder IRD/ORD actual values.

   initiator (First RDMA Message) -->  After the initiator modifies its
      ORD/IRD to match the responder's values as stated above, the
      initiator sends the first message of negotiated RTR indication
      option.  If no matching RTR indication option exists then the
      initiator sends a TERM message.

   The initiator or responder MUST generate the TERM message that
   contains Layer 2, Error Type 0, Error Code 5 when it encounters any
   error locally for which the special Error Code is not defined in
   Section 8 before resetting the connection.




Kanevsky, et al.          Expires June 17, 2012                [Page 20]

Internet-Draft   Enhanced RDMA Connection Establishment    December 2011


10.  Interoperability

   The initiator requests enhanced RDMA connection establishment by
   sending an enhanced RDMA establishment request; an enhanced responder
   is REQUIRED to respond with an enhanced RDMA connection establishment
   response, whereas an unenhanced responder treats the enhanced request
   as incorrectly formatted and closes the TCP connection.  All
   responders are REQUIRED to issue unenhanced RDMA connection
   establishment responses in response to unenhanced RDMA connection
   establishment requests.

   The initiator MUST NOT use the enhanced RDMA connection establishment
   formats or function codes when no enhanced functionality is desired.

   The responder MUST continue to accept unenhanced connection requests.

   There are three initiator/responder cases that involve enhanced MPA:
   both the initiator and responder, only the responder, and only the
   initiator.  The enhanced MPA frame is defined by field 'S' set to 1.

   Enhanced MPA initiator and responder:  If the responder receives an
      enhanced MPA message, it MUST respond with an enhanced MPA
      message.

   Enhanced MPA responder only:  If the responder receives an unenhanced
      MPA message ('S' is set to 0), it MUST respond with an unenhanced
      MPA message.

   Enhanced MPA initiator only:  If the responder receives an enhanced
      MPA message and it does not support enhanced RDMA connection
      establishment, it MUST close the TCP connection and exit MPA.
      From a standard RDMA connection establishment point of view
      enhanced MPA frame is improperly formatted as stated in [RFC5044].
      Thus, both the initiator and responder report TCP connection
      termination to an application locally.  In this case the initiator
      MAY attempt to establish an RDMA connection using the unenhanced
      MPA protocol as defined in [RFC5044] if this protocol is
      compatible with the application, and let ULP deal with ORD and
      IRD, and peer-to-peer negotiations.

   A note for a potential future enhancements for connection
   establishment negotiation: It is possible to further extend
   formatting of private data of the MPA Request and Reply frames and to
   use other bits from "Res" field to indicate additional private data
   formatting.






Kanevsky, et al.          Expires June 17, 2012                [Page 21]

Internet-Draft   Enhanced RDMA Connection Establishment    December 2011


11.  IANA Considerations

   IANA is requested to add the following entries to the "SCTP Function
   Codes for DDP Session Control" registry created by Section 3.4 of
   [IANA_RDDP_REGISTRY]:

   0x0005,  Enhanced DDP Stream Session Initiate, [RFCXXXX]

   0x0006,  Enhanced DDP Stream Session Accept, [RFCXXXX]

   0x0007,  Enhanced DDP Stream Session Reject, [RFCXXXX]

   IANA is requested to add the following entries to the "MPA Errors"
   registry created by Section 3.3 of [IANA_RDDP_REGISTRY]

   0x2/0x0/0x05,  - MPA Error / Local catastrophic error, [RFCXXXX]

   0x2/0x0/0x06  - MPA Error / Insufficient IRD resources, [RFCXXXX]

   0x2/0x0/0x07  - MPA Error / No matching RTR option, [RFCXXXX]

   RFC Editor: Please replace XXXX in the six instances of [RFCXXXX]
   above with the RFC number of this document and remove this note.


12.  Security Considerations

   The security considerations from RFC 5044 and RFC 5043 apply and the
   changes in this document do not introduce new security
   considerations.  However it is recommended that implementations do
   sanity checking for the input parameters, including ORD, IRD, and the
   control flags used for RTR indication option negotiation.


13.  Acknowledgements

   The authors wish to thank Sean Hefty, Dave Minturn, Tom Talpey, David
   Black and David Harrington for their valuable contributions and
   reviews of this document.


14.  References

14.1.  Normative References

   [IANA_RDDP_REGISTRY]
              "IANA Registries for the RDDP (Remote Direct Data
              Placement) Protocols, Work in Progress", October, 2011, <h



Kanevsky, et al.          Expires June 17, 2012                [Page 22]

Internet-Draft   Enhanced RDMA Connection Establishment    December 2011


              ttp://www.ietf.org/internet-drafts/
              draft-ietf-storm-rddp-registries-00.txt>.

   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
              Requirement Levels", BCP 14, RFC 2119, March 1997.

   [RFC4960]  Stewart, R., "Stream Control Transmission Protocol",
              RFC 4960, September 2007.

   [RFC5040]  Recio, R., Metzler, B., Culley, P., Hilland, J., and D.
              Garcia, "A Remote Direct Memory Access Protocol
              Specification", RFC 5040, October 2007.

   [RFC5041]  Shah, H., Pinkerton, J., Recio, R., and P. Culley, "Direct
              Data Placement over Reliable Transports", RFC 5041,
              October 2007.

   [RFC5043]  Bestler, C. and R. Stewart, "Stream Control Transmission
              Protocol (SCTP) Direct Data Placement (DDP) Adaptation",
              RFC 5043, October 2007.

   [RFC5044]  Culley, P., Elzur, U., Recio, R., Bailey, S., and J.
              Carrier, "Marker PDU Aligned Framing for TCP
              Specification", RFC 5044, October 2007.

14.2.  Informative References

   [DAPL]     "Direct Access Programming Library",
              <http://www.datcollaborative.org>.

   [IBTA]     "InfiniBand Architecture Specification Release 1.2.1", <ht
              tp://www.infinibandta.org/content/
              pages.php?pg=technology_overview>.

   [OFA]      "OFA verbs & APIs", <http://www.openfabrics.org/>.

   [OpenMP]   McGraw-Hill, "Parallel Programming in C with MPI and
              OpenMP", 2003.

   [PPMPI]    Morgan Kaufmann Publishers Inc., "Parallel Programming
              with MPI", 2008.

   [RDMAC]    "RDMA Protocol Verbs Specification (Version 1.0)", <http:/
              /www.rdmaconsortium.org/home/
              draft-hilland-iwarp-verbs-v1.0-RDMAC.pdf>.

   [RDS]      Open Fabrics Association, "Reliable Datagram Socket",
              2008, <http://www.openfabrics.org/archives/



Kanevsky, et al.          Expires June 17, 2012                [Page 23]

Internet-Draft   Enhanced RDMA Connection Establishment    December 2011


              spring2008sonoma/Tuesday/sonoma_2008_0408%20Oracle.ppt>.

   [UsingMPI]
              MIT Press, "Using MPI-2: Advanced Features of the Message
              Passing Interface", 1999.

   [VIA]      Compaq, Intel, Microsoft, "Virtual Interface Architecture
              Specification", 1997, <http://pllab.cs.nthu.edu.tw/cs5403/
              Readings/EJB/san_10.pdf>.


Authors' Addresses

   Arkady Kanevsky (editor)
   Dell Inc.
   One Dell Way, MS PS2-47
   Round Rock, TX  78682
   USA

   Phone: +1-512-728-0000
   Email: arkady.kanevsky@gmail.com


   Caitlin Bestler (editor)
   Nexenta Systems
   555 E El Camino Real #104
   Sunnyvale, CA  94087
   USA

   Phone: +1-949-528-3085
   Email: Caitlin.Bestler@nexenta.com


   Robert Sharp
   Intel
   LAD High Performance Message Passing, Mailstop: AN1-WTR1
   1501 South Mopac, Suite 400
   Austin, TX  78746
   USA

   Phone: +1-512-493-3242
   Email: robert.o.sharp@intel.com









Kanevsky, et al.          Expires June 17, 2012                [Page 24]

Internet-Draft   Enhanced RDMA Connection Establishment    December 2011


   Steve Wise
   Open Grid Computing
   4030 Braker Lane STE 130
   Austin, TX  78759
   USA

   Phone: +1-512-343-9196 x101
   Email: swise@opengridcomputing.com











































Kanevsky, et al.          Expires June 17, 2012                [Page 25]

