Network Working Group                                     David Melman
Internet Draft                                             Tal Mizrahi
Intended status: Informational                                 Marvell
Expires: March 2012                                    Donald Eastlake
                                                                Huawei
                                                    September 11, 2011

                              FCoE over TRILL
                        draft-mme-trill-fcoe-01.txt


Abstract

   Fibre Channel over Ethernet (FCoE) and TRILL are two emerging
   standards in the data center environment. While these two protocols
   are seemingly unrelated, they have a very similar behavior in the
   forwarding plane, as both perform hop-by-hop forwarding over
   Ethernet, modifying the packet's MAC addresses at each hop. This
   document describes an architecture for the integrated deployment of
   these two protocols.

Status of this Memo

   This Internet-Draft is submitted to IETF in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF), its areas, and its working groups.  Note that
   other groups may also distribute working documents as Internet-
   Drafts.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   The list of current Internet-Drafts can be accessed at
   http://www.ietf.org/ietf/1id-abstracts.txt.

   The list of Internet-Draft Shadow Directories can be accessed at
   http://www.ietf.org/shadow.html.

   This Internet-Draft will expire on March 11, 2012.

Copyright Notice

   Copyright (c) 2011 IETF Trust and the persons identified as the
   document authors. All rights reserved.



Melman, et al.         Expires March 11, 2012                 [Page 1]

Internet-Draft             FCoE over TRILL              September 2011


   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents
   (http://trustee.ietf.org/license-info) in effect on the date of
   publication of this document. Please review these documents
   carefully, as they describe your rights and restrictions with respect
   to this document. Code Components extracted from this document must
   include Simplified BSD License text as described in Section 4.e of
   the Trust Legal Provisions and are provided without warranty as
   described in the Simplified BSD License.

Table of Contents

   1. Introduction ................................................ 2
   2. Abbreviations ............................................... 3
   3. FCoE over TRILL ............................................. 4
      3.1. FCoE over a TRILL Cloud................................. 4
      3.2. FCoE over RBridge....................................... 5
         3.2.1. FCRB .............................................. 5
         3.2.2. Topology .......................................... 7
         3.2.3. The FCRB Flow...................................... 8
   4. Security Considerations..................................... 10
   5. IANA Considerations ........................................ 10
   6. Acknowledgments ............................................ 11
   7. References ................................................. 11
      7.1. Normative References................................... 11
      7.2. Informative References................................. 11


1. Introduction

   Data center networks are rapidly evolving towards a consolidated
   approach, where Ethernet is used as the common infrastructure for all
   types of traffic. Storage traffic, which was traditionally dominated
   by the Fibre Channel (FC) protocol suite, is evolving towards Fibre
   Channel over Ethernet (FCoE), where native FC packets are
   encapsulated with an FCoE encapsulation over an Ethernet header.

   Traffic between two FCoE end nodes (ENodes) is forwarded through one
   or more FCoE Forwarders (FCF). An FCF takes a forwarding decision
   based on the Fibre Channel destination ID (D_ID), and enforces
   security policies between ENodes, also known as zoning. Once an FCF
   takes a forwarding decision, it modifies the source and destination
   MAC addresses of the packet, to reflect the path to the next hop FCF
   or ENode. FCFs use a routing protocol called Fabric Shortest Path
   First (FSPF) to find the optimal path to each destination. An FCF
   typically has one or more native Fibre Channel interfaces, allowing



Melman, et al.         Expires March 11, 2012                 [Page 2]

Internet-Draft             FCoE over TRILL              September 2011


   it to communicate with native Fibre Channel devices, e.g., storage
   arrays.

   TRILL [RFCTRILL] is a protocol for transparent least cost routing,
   where RBridges forward traffic to their detination based on a least
   cost route, using a TRILL encapsulation header. RBridges forward
   TRILL-encapsulated packets based on the Egress RBridge Nickname in
   the TRILL header. An RBridge forwards a TRILL-encapsulated packet
   after modifying its MAC addresses to reflect the path to the next-hop
   RBridge, and decrementing a Hop Count field.

   TRILL and FCoE bear a strong resemblance in their forwarding planes.
   Both protocols take a forwarding decision based on protocol addresses
   above Layer 2, and modify the Ethernet MAC addresses on a per-hop
   basis. Each of the protocols uses its own routing protocol rather
   than using any type of bridging protocol such as spanning tree
   protocol [802.1Q] or the Shortest Path Bridging protocol [802.1aq].

   FCoE and TRILL are both targeted at the data center environment, and
   their concurrent deployment is self-evident. This document describes
   an architecture for the integrated deployment of these two protocols.

2. Abbreviations

   ENode   FCoE Node such as server or storage array

   EoR    End of Row

   FC     Fibre Channel

   FCF    Fibre Channel Forwarder

   FCoE   Fibre Channel over Ethernet

   FCRB   Fibre Channel forwarder over RBridge

   FSPF   Fabric Shortest Path First

   LAN    Local Area Network

   RBridge Routing Bridge

   SAN    Storage Area Network

   ToR    Top of Rack

   TRILL   Transparent Interconnection of Lots of Links


Melman, et al.         Expires March 11, 2012                 [Page 3]

Internet-Draft             FCoE over TRILL              September 2011


   WAN    Wide Area Network

3. FCoE over TRILL

3.1. FCoE over a TRILL Cloud

   The simplest approach for running FCoE traffic over a TRILL network
   is presented in Figure 1. The figure illustrates a TRILL-enabled
   network, where FCoE traffic is transparently forwarded over the TRILL
   cloud. The figure illustrates two ENodes, a Server and an FCoE
   Storage Array, an FCF, and a native Fibre Channel SAN connected to
   the FCF.

   FCoE traffic between the two ENodes is sent from the first ENode over
   the TRILL cloud to the FCF, and then back through the TRILL cloud to
   the second ENode.

            +---+
            |   |_________
            |   |         \  ___   _
            +---+          \/   \_/ \__                  _   __
         FCoE Storage     _/           \                / \_/  \_
            Array        /    TRILL    /       +---+    \_       \
          (ENode A)      \_   Cloud   /________|   |____/  SAN  _/
                          /           \        |   |    \__   _/
                          \__/\_   ___/        +---+       \_/
            +---+         /     \_/             FCF
            |   |________/
            |   |
            +---+
            Server
          (ENode B)
                  Figure 1 The "Separate Cloud" Approach

   The configuration in Figure 1 separates the TRILL cloud(s) and the
   FCoE cloud(s). The TRILL cloud forwards FCoE traffic as standard
   Ethernet traffic, and appears to the ENodes and FCF as an Ethernet
   LAN.

   The main drawback of the Separate Cloud approach is that RBridges and
   FCFs are separate nodes in the network, resulting in more cabeling
   and boxes, and communication between Enodes usually requires two
   TRILL cloud traversals with twice as many hops. As mentioned above,
   data center networking is converging towards a consolidated and cost
   effective approach, where the same infrastructure and equipment is


Melman, et al.         Expires March 11, 2012                 [Page 4]

Internet-Draft             FCoE over TRILL              September 2011


   used for both data and storage traffic, and where high efficiency and
   minimal number of hops are important factors when designing the
   network topology.

3.2. FCoE over RBridge

3.2.1. FCRB

   Rather than the Separate Cloud approach discussed in the previous
   subsection, an alternate approach is presented, where each switch
   incorporates both an FCF entity and an RBridge entity. This
   consolidated entity is referred to as FCoE-forwarder-over-RBridge
   (FCRB).

   Figure 2 illustrates an FCRB, and its main building blocks. An FCRB
   can be functionally viewed as two independent entities:

   o An FCoE Forwarder (FCF) entity.

   o An RBridge entity.

   The FCF entity is connected to one of the ports of the RBridge, and
   appears to the RBridge as a native Ethernet host. A detailed
   description of the interaction between the layers is presented in
   Section 3.2.3.























Melman, et al.         Expires March 11, 2012                 [Page 5]

Internet-Draft             FCoE over TRILL              September 2011


                       +--------------------+
                       |FCRB                |
                       |   +-----------+    |
                       |   |    FCF    |-+  |
                       |   +-----+-----+ |  |
                       |         |       |  |
                       |   +-----+-----+ |  |
                       |   |  RBridge  | |  |
                       |   +-+-+-+-+-+-+ |  |
                       |   | | | | | |   |  |
                       +---|-|-|-|-|-|---|--+               _   __
              FCoE/        / | | | | |    \        Native  / \_/  \_
   +---+    Ethernet      /  / | | | |     \         FC    \_       \
   |   |_________________/  /  | | | |      \______________/  SAN  _/
   |   |                   /   | | | |                     \__   _/
   +---+                  /    | | | |                        \_/
FCoE Storage             /     | | | | FCoE / Ethernet
   Array                /      |_| | | over TRILL
 (ENode A)             /      /   \_/ \__
                      /     _/           \
   +---+             /     /    TRILL    /
   |   |____________/      \_   Cloud   /
   |   |                    /           \
   +---+                    \__/\_   ___/
   Server                         \_/
 (ENode B)
                    Figure 2 FCRB Entity in the Network

   The FCRB entity maintains layer independence between the TRILL and
   FCoE protocols, while enabling both protocols on the same network.

   It is noted that FCoE traffic is always forwarded through an FCF, and
   cannot be forwarded directly between two ENodes. Thus, FCoE traffic
   between ENodes A and B in the topology in Figure 1 is forwarded
   through the path

   ENode A-->TRILL cloud-->FCF-->TRILL cloud-->ENode B

   Traffic between A and B in the topology in Figure 2 is forwarded
   through the path

   ENode A-->FCRB-->ENode B



Melman, et al.         Expires March 11, 2012                 [Page 6]

Internet-Draft             FCoE over TRILL              September 2011


   Hence, the usage of FCRB entities allows TRILL and FCoE to use common
   infrastructure and equipment, as opposed to the Separate Cloud
   topology presented in Figure 1.

3.2.2. Topology

   The network configuration illustrated in Figure 3 shows a typical
   topology of a data center network. Servers are hierarchically
   connected through Top-of-Rack (ToR) switches, and End-of-Row (EoR)
   racks. The EoR switches to other clouds, such as an external WAN, or
   a native FC SAN.





































Melman, et al.         Expires March 11, 2012                 [Page 7]

Internet-Draft             FCoE over TRILL              September 2011


                        _   __               _   __
                       / \_/  \_            / \_/  \_
                       \_       \           \_       \    .....
                       /  SAN  _/          _/  WAN  _/
                       \__   _/  \        / \__   _/
                          \_/     \      /     \_/
                           |       \    /        |
                           |        \  /         |
                           |         \/          |
       EoR              +----+_______/\_______+----+
       FCoE over        |    |                |    |
       RBridge          |    |                |    |
       (FCRB)           +----+                +----+
                        /    \                /    \
                       /      \              /      \
       ToR         +---+      +---+      +---+      +---+
       FCoE over   |   |      |   |      |   |      |   |
       RBridge     |   |      |   |      |   |      |   |
       (FCRB)      +---+      +---+      +---+      +---+
                    / \        / \        / \        / \
                   /   \      /   \      /   \      /   \
                 +-+   +-+  +-+   +-+  +-+   +-+  +-+   +-+
       Servers/  | |   | |  | |   | |  | |   | |  | |   | |
       ENodes    +-+   +-+  +-+   +-+  +-+   +-+  +-+   +-+
                  A     B    C     D    E     F    G     H

                    Figure 3 FCoE over RBridge Topology

   Note that in the example in Figure 3 all the ToR and EoR switches are
   FCRB entities, but it is also possible for some of the network nodes
   to be pure RBridges, creating a topology where FCRBs are
   interconnected through TRILL clouds.

3.2.3. The FCRB Flow

   FCoE traffic sent between two ENodes, A and B, is transmitted through
   the ToR FCRB, since A and B are connected to the same ToR. Traffic
   between A and C must be forwarded through the EoR FCRB.







Melman, et al.         Expires March 11, 2012                 [Page 8]

Internet-Draft             FCoE over TRILL              September 2011


+--------+     +--------+     +--------+     +--------+     +--------+
| FCoE   |.....|  FCF   |.....|  FCF   |.....|  FCF   |.....| FCoE   |
| ENode  |     +--------+     +--------+     +--------+     | ENode  |
|        |     |RBridge |.....|RBridge |.....|RBridge |     |        |
+--------+     +--------+     +--------+     +--------+     +--------+
|Ethernet|<===>|Ethernet|<===>|Ethernet|<===>|Ethernet|<===>|Ethernet|
+--------+     +--------+     +--------+     +--------+     +--------+
  Server          ToR            EoR            ToR        FCoE Storage
  ENode A         FCRB           FCRB           FCRB          Array
                                                             ENode C
               Figure 4 Traffic between two ENodes - Example

   Figure 4 illustrates the traffic between ENodes A and C that are not
   connected to the same ToR.

   o FCoE traffic from A is sent to the ToR over the Ethernet
      interface.

   o The RBridge entity at the ToR forwards the packet to the FCF,
      analogous to forwarding between two Ethernet hosts.
      The FCF entity at the ToR takes a forwarding decision, and updates
      the destination MAC address of the packet to the address of the
      EoR FCF. The packet is then forwarded to the RBridge entity, where
      it is encapsulated in a TRILL header, and sent to the EoR FCRB.

   o The RBridge entity in the EoR FCRB, acting as the egress RBridge,
      decapsulates the TRILL header and forwards the FCoE packet to the
      FCF entity. The FCF takes a forwarding decision and updates the
      MAC address of the packet according to the next hop ToR. The
      packet is then forwarded to the RBridge and encapsulated with a
      new TRILL header.

   o At the ToR FCRB, the packet reaches the final egress RBridge, and
      the TRILL encapsulation is removed. The FCF then forwards the
      packet to the RBridge entity after updating its MAC addresses. The
      RBridge entity forwards the packet to the target ENode.











Melman, et al.         Expires March 11, 2012                 [Page 9]

Internet-Draft             FCoE over TRILL              September 2011


         +--------+     +--------+     +---------+     +--------+
         |  FCoE  |.....|  FCF   |.....|   FCF   |.....|   FC   |
         |  ENode |     +--------+     +----+----+     |protocol|
         |        |     |RBridge |.....| RB |    |     | stack  |
         +--------+     +--------+     +----+ FC |     |        |
         |Ethernet|<===>|Ethernet|<===>|Eth |    |<===>|        |
         +--------+     +--------+     +----+----+     +--------+
           Server          ToR            EoR          Native FC
           ENode           FCRB           FCRB       Storage Array

      Figure 5 Example Traffic between ENode & Native FC Storage Array

   Figure 5 illustrates traffic sent between an ENode and an FC Storage
   Array, following the network topology in Figure 3.

   o FCoE traffic from the ENode is sent to the ToR over the Ethernet
      interface.

   o The RBridge entity at the ToR forwards the packet to the FCF. The
      FCF entity at the ToR takes a forwarding decision and updates the
      destination MAC address of the packet to the address of the EoR
      FCF. The packet is then forwarded to the RBridge entity, where it
      is encapsulated in a TRILL header, and sent to the EoR FCRB.

   o The egress RBridge entity at the EoR FCRB decapsulates the TRILL
      header, and forwards the FCoE packet to the FCF entity. The packet
      is then forwarded as a native FC packet through the FC interface
      to the native FC node.



4. Security Considerations

   For general TRILL Security Considerations see [RFCTRILL].

   For general FCoE Security Consideration see Annex D of [FC-BB-5].

   There are no additional security implications imposed by this
   document.

5. IANA Considerations

   There are no IANA actions required by this document.

   RFC Editor: please delete this section before publication.


Melman, et al.         Expires March 11, 2012                [Page 10]

Internet-Draft             FCoE over TRILL              September 2011


6. Acknowledgments

   This document was prepared using 2-Word-v2.0.template.dot.

7. References

7.1. Normative References

   [RFCTRILL]    Perlman, R., Eastlake, D., Dutt, D., Gai, S.,
                 Ghanwani, A., "Routing Bridges (RBridges): Base
                 Protocol Specification", RFC6325, July 2011.

7.2. Informative References

   [FC-BB-5]     ANSI INCITS 462: Information Technology - Fibre
                 Channel - Backbone - 5 (FC-BB-5).

   [802.1Q]      "IEEE Standard for Local and metropolitan area networks
                 - Virtual Bridged Local Area Networks", IEEE Std
                 802.1Q-2011, May 2011.

   [802.1aq]     "IEEE Standard for Local and metropolitan area
                 networks - Shortest Path Bridging", work in progress,
                 June 2011.

Authors' Addresses

   David Melman
   Marvell
   6 Hamada St.
   Yokneam, 20692 Israel

   Email: davidme@marvell.com



   Tal Mizrahi
   Marvell
   6 Hamada St.
   Yokneam, 20692 Israel

   Email: talmi@marvell.com


   Donald Eastlake 3rd
   Huawei Technologies



Melman, et al.         Expires March 11, 2012                [Page 11]

Internet-Draft             FCoE over TRILL              September 2011


   155 Beaver Street
   Milford, MA 01757 USA

   Phone: +1-508-333-2270
   EMail: d3e3e3@gmail.com











































Melman, et al.         Expires March 11, 2012                [Page 12]

