Application Centric Infrastructure (ACI) Fabric Initialization

How to Connect APICs to the Cisco ACI Fabric

✍️ Written by: RJS Expert | RJS Cloud Academy
A comprehensive guide to Cisco ACI fabric initialization, APIC configuration, and fabric discovery process.

📋 ACI Fabric Setup Overview

To setup the Application Centric Infrastructure (ACI) Fabric, the following tasks need to be completed:

• ✅ Rack and Cable the Hardware
• ✅ Configure each Cisco APIC's Integrated Management Controller (CIMC)
• ✅ Check APIC firmware and software versions
• ✅ Check the image type (NX-OS/Cisco ACI) and software version of switches
• ✅ APIC1 initial setup
• ✅ Fabric discovery
• ✅ Setup the remainder of APIC Cluster

---

1. Rack and Cable the Hardware

APIC Connectivity

The APICs will be connected to Leaf switches. When using multiple APICs, we recommend connecting APICs to separate Leafs for redundancy purposes.

⚠️ Important Note for APIC M3/L3:

If it's APIC M3/L3, VIC 1445 has four ports (port-1, port-2, port-3, and port-4 from left to right):

• Port-1 and port-2 make a single pair corresponding to eth2-1 on the APIC
• Port-3 and port-4 make another pair corresponding to eth2-2 on the APIC
• Only a single connection is allowed for each pair
• All ports must be configured for the same speed, either 10G or 25G

Switch Connectivity

All Leaf switches will need to connect to spine switches and vice versa. This provides your fabric with a fully redundant switching fabric.

In addition to the fabric network connections, you'll also connect:

• 🔌 Redundant PSUs to separate power sources
• 🌐 Management Interface to your 1G out-of-band management network
• 💻 Console connection to a Terminal server (optional, but highly recommended)

---

2. Configure Cisco APIC Integrated Management Controller (CIMC)

When you first connect your CIMC connection marked with "mgmt." on the rear facing interface, it will be configured for DHCP by default. Cisco recommends that you assign a static address to avoid any loss of connectivity or changes to address leases.

Configuring CIMC via Console

You can modify the CIMC details by connecting a crash cart (physical monitor, USB keyboard and mouse) to the server and powering it on. During the boot sequence, it will prompt you to press "F8" to configure the CIMC.

NIC Mode Configuration

Dedicated Mode (Recommended):

• ✅ Utilizes the dedicated "mgmt." interface in the rear of the APIC appliance
• ✅ Separates CIMC platform management traffic
• ✅ Prevents fabric discovery issues

Shared LOM Mode (Not Recommended):

• ❌ Sends CIMC traffic over the LAN on Motherboard (LOM) port
• ❌ Shares bandwidth with APIC OS management traffic
• ❌ Can cause issues with fabric discovery if not properly configured

Aside from the IP address details, the rest of the options can be left alone unless there's a specific reason to modify them. Once a static address has been configured you will need to Save the settings & reboot.

Logging into the CIMC Web Interface

After a few minutes you should be able to reach the CIMC Web Interface using the newly assigned IP along with the default CIMC credentials:

Default Credentials:
• Username: admin
• Password: password

⚠️ It's recommended that you change the CIMC default admin password after first use.

To log into the CIMC, open a web browser to https://<CIMC_IP>. You'll need to ensure you have flash installed & permitted for the URL.

⚠️ Note: Launching the KVM console will require that you have Java version 1.6 or later installed. Depending on your client security settings, you may need to whitelist the CIMC address within your local Java settings for the KVM applet to load.

---

3. Check APIC Firmware and Software

All your APICs require to run the same version when joining a cluster. This may require manually upgrading/downgrading your APICs prior to joining them to the fabric.

📚 Reference: Instructions on upgrading standalone APICs using KVM vMedia can be found in the Cisco APIC Management, Installation, Upgrade, and Downgrade Guide for your respective version.

Switch nodes can be running any version of ACI switch image and can be upgraded/downgraded once joined to the fabric via firmware policy.

---

4. Check Image Type (NX-OS vs Cisco ACI)

For a Nexus 9000 series switch to be added to an ACI fabric, it needs to be running an ACI image. Switches that are ordered as "ACI Switches" will typically be shipped with an ACI image.

Converting NX-OS to ACI Mode

If you have existing standalone Nexus 9000 switches running traditional NX-OS, then you may need to install the appropriate image (For example, aci-n9000-dk9.14.0.1h.bin).

📚 Reference: See the Cisco Nexus 9000 Series NX-OS Software Upgrade and Downgrade Guide on CCO for your respective version of NX-OS.

---

5. APIC1 Initial Setup

Now that you have basic remote connectivity, you can complete the setup of your ACI fabric from any workstation with network access to the APIC. If the server is not powered on, do so now from the CIMC interface. The APIC will take 3-4 minutes to fully boot.

Next, open a console session via the CIMC KVM console. Assuming the APIC has completed the boot process, it should be sitting at a prompt "Press any key to continue…". Doing so will begin the setup utility.

Setup Dialogue Parameters

From here, the APIC will guide you through the initial setup dialogue. Carefully answer each question. Some of the items configured can't be changed after initial setup, so review your configuration before submitting it.

Configuration Parameters Explained

Parameter	Description
Fabric Name	User defined, will be the logical friendly name of your fabric.
Fabric ID	Leave this ID as the default 1.
Number of Controllers	Set this to the number of APICs you plan to configure. This can be increased/decreased later.
Pod ID	The Pod ID to which this APIC is connected to. If this is your first APIC or you don't have more than a single Pod, this will always be 1.
Standby Controller	Beyond your active controllers (typically 3) you can designate additional APICs as standby. In the event you have an APIC failure, you can promote a standby to assume the identity of the failed APIC.
APIC-X	A special-use APIC model used for telemetry and other heavy ACI App purposes. For your initial setup this typically would not be applicable. (May be referenced as "ACI Services Engine" in future releases)

Critical Configuration Parameters (Cannot Be Changed)

⚠️ TEP Pool (Tunnel Endpoint Pool)

This will be a subnet of addresses used for internal fabric communication. This subnet will NOT be exposed to your legacy network unless you're deploying the Cisco AVS or Cisco ACI Virtual Edge.

Recommendation: Assign an unused subnet of size between /16 and /21. Most customers allocate an unused /16 and move on. The size of the subnet used will impact the scale of your Pod.

⛔ Important: This value CANNOT be changed once configured. Having to modify this value requires a wipe of the fabric.

🚫 Subnet Restriction: The 172.17.0.0/16 subnet is NOT supported for the infra TEP pool due to a conflict with the docker0 interface. If you must use this subnet, you must manually configure the docker0 IP address to be in a different address space in each Cisco APIC before clustering.

⚠️ Infra VLAN

This is the VLAN ID for all fabric connectivity. This VLAN ID should be allocated solely to ACI, and not used by any other legacy device in your network.

Though this VLAN is used for fabric communication, there are certain instances where this VLAN ID may need to be extended outside of the fabric such as the deployment of the Cisco AVS/AVE.

Recommendation: Cisco recommends a VLAN smaller than VLAN 3915 as being a safe option as it is not a reserved VLAN on Cisco DC platforms as of today.

⛔ Important: This value CANNOT be changed once configured. Having to modify this value requires a wipe of the fabric.

⚠️ BD Multicast Pool (GIPO)

Used for internal connectivity. We recommend leaving this as the default or assigning a unique range not used elsewhere in your infrastructure.

⛔ Important: This value CANNOT be changed once configured. Having to modify this value requires a wipe of the fabric.

Submitting the Configuration

Once the Setup Dialogue has been completed, it will allow you to review your entries before submitting. If you need to make any changes enter "y" otherwise enter "n" to apply the configuration.

After applying the configuration allow the APIC 4-5 minutes to fully bring all services online and initialize the REST login services before attempting to login through a web browser.

---

6. Fabric Discovery

With our first APIC fully configured, now we will login to the GUI and complete the discovery process for our switch nodes.

When logging in for the first time, you may have to accept the certificate warnings and/or add your APIC to the exception list.

Registering Fabric Nodes

Navigate to: Fabric tab → Inventory sub-tab → Fabric Membership folder

From this view you are presented with your registered fabric nodes. Click on the Nodes Pending Registration tab in the work pane and we should see our first Leaf switch waiting discovery.

⚠️ Note: This would be one of the Leaf switches where the APIC is directly connected to.

To register our first node:

1. Click on the first row
2. From the Actions menu (Tool Icon) select Register

Node Registration Details

The Register wizard will require some details including:

Field	Description
Node ID	The unique identifier you wish to assign. Cannot be changed once assigned without decommissioning. Common Practice: • Leaf switches: 100+ • Spine switches: 200+
Node Name	Hostname for the switch. Can be modified later.
RL TEP Pool	Reserved for Remote Leafs usage only. Doesn't apply to local fabric-connected Leaf switches.
Rack Name	Optional field for organizational purposes.

This information is provided to the APIC via LLDP TLVs. If a switch was previously registered to another fabric without being erased, it would never appear as an unregistered node. It's important that all switches have been wiped clean prior to discovery.

Node Bootstrap Process

Once the registration details have been submitted, the entry for this leaf node will move from the Nodes Pending Registration tab to the Registered Nodes tab under Fabric Membership.

The node will take 3 to 4 minutes to complete the discovery, which includes the bootstrap process and bringing the switch to an "Active" state. During the process, you will notice a tunnel endpoint (TEP) address gets assigned from your Infra TEP pool (such as 10.0.0.0/16).

In-Depth: Fabric Discovery Process

📡 Discovery Process Steps

1. LLDP Neighbor Discovery: Cisco APIC uses LLDP to discover a switch
2. DHCP Request: After successful discovery, the switch sends a request for an IP address via DHCP
3. TEP Address Allocation: Cisco APIC allocates an address from the DHCP pool. The switch uses this address as a TEP address
4. Boot File Download: In the DHCP Offer packet, Cisco APIC passes the boot file information. The switch retrieves this file via HTTP GET to port 7777 of Cisco APIC
5. Firmware Loading: The boot file HTTP GET 200 OK response contains the firmware that the switch will load
6. IFM Establishment: Cisco APIC initiates the encrypted TCP session when the switch is listening on TCP port 12183 to establish the policy element Intra-Fabric Messaging (IFM)

🔐 IFM Security

Communication between the various nodes and processes in the Cisco ACI Fabric uses IFM (Intra-Fabric Messaging), and IFM uses SSL-encrypted TCP communication.

Each Cisco APIC and fabric node has 1024-bit SSL keys that are embedded in secure storage. The SSL certificates are signed by Cisco Manufacturing Certificate Authority (CMCA).

A fabric node is considered active when the Cisco APIC and the node can exchange heartbeats through the IFM process.

Node Status States

Node status may fluctuate between several states during the fabric registration process. The states are shown in the Fabric Node Vector table. The APIC CLI command to show the Fabric Node Vector table is acidiag fnvread.

State	Description
Unknown	Node discovered but no Node ID policy configured
Undiscovered	Node ID configured but not yet discovered
Discovering	Node discovered but IP not yet assigned
Unsupported	Node is not a supported model
Disabled	Node has been decommissioned
Inactive	No IP connectivity
Active ✓	Node is fully operational

💡 Note: ACI uses inter-fabric messaging (IFM) packets to communicate between the different nodes or between leaf and spine. These IFM packets are typically TCP packets, which are secured by 1024-bit SSL encryption, and the keys used for encryption are stored on secure storage. These keys are signed by Cisco Manufacturing Certificate Authority (CMCA). Any issues with IFM process can prevent fabric nodes communicating and from joining the fabric.

Completing Node Registration

After the first Leaf has been discovered and moved to an Active state, it will then discover every Spine switch it's connected to. Go ahead and register each Spine switch in the same manner.

Since each Leaf Switch connects to every Spine switch, once the first Spine completes the discovery process, you should see all remaining Leaf switches pending registration. Go ahead with registering all remaining nodes and wait for all switches to transition to an Active state.

---

7. Setup Remainder of APIC Cluster

With all the switches online & active, our next step is to finish the APIC cluster configuration for the remaining nodes.

Navigate to: System → Controllers sub menu → Controllers Folder → apic1 → Clusters as Seen by this Node folder

From here you will see your single APIC along with other important details such as the Target Cluster Size and Current Cluster Size. Assuming you configured apic1 with a cluster size of 3, we'll have two more APICs to setup.

Adding APIC2 and APIC3

At this point we would want to now open the KVM console for APIC2 and begin running through the setup Dialogue just as we did for APIC1 previously.

⚠️ Critical Requirements for Additional APICs

When joining additional APICs to an existing cluster it's imperative that you configure:

• ✅ Same Fabric Name
• ✅ Same Infra VLAN
• ✅ Same TEP Pool
• ✅ Unique Controller ID (set to ID 2 for APIC2, ID 3 for APIC3)

You'll notice that you will not be prompted to configure Admin credentials. This is expected as they will be inherited from APIC1 once you join the cluster.

Allow APIC2 to fully boot and bring its services online. You can confirm everything was successfully configured as soon as you see the entry for APIC2 in the Active Controllers view.

During this time, it will also begin syncing with APIC1's configuration. Allow 4-5 minutes for this process to complete. During this time, you may see the State of the APICs transition back & forth between Fully Fit and Data Layer Synchronization in Progress.

Continue through the same process for APIC3, ensuring you assign the correct controller ID.

🎉 Fabric Discovery Complete!

This concludes the entire fabric discovery process. All your switches & controllers will now be in sync and under a single pane of management.

Your ACI fabric can be managed from any APIC IP. All APICs are active and maintain a consistent operational view of your fabric.

---

Complete IFM (Intra-Fabric Messaging) Process

After all processes are completed, the fabric is ready for production configuration. The complete IFM steps are:

🔄 IFM Process Steps

1. Link Layer Discovery Protocol (LLDP) Neighbor Discovery
2. Tunnel End Point (TEP) IP address assignment to the node via DHCP
3. Node software upgraded if necessary
4. ISIS adjacency mode
5. Certification Validation
6. Start of DME Process on switches
7. Tunnel Setup (iVXLAN)
8. Policy Element IFM Setup

---

Fabric Initialization Task Checklist

✅ Post-Setup Tasks

• ✅ Configure APIC1
• ✅ Add first Leaf to fabric
• ✅ Add all Spines to fabric
• ✅ Add remaining Leafs to fabric
• ✅ Add remaining APICs to fabric
• 🔧 Setup NTP
• 🔧 Configure OOB Management IP Pool
• 🔧 Configure Export Policies for Configuration and Tech Support Exports
• 🔧 Configure Firmware Policies (For Upgrades)

🚀 Your ACI Fabric is Now Ready!

You can now proceed with tenant configuration, EPG creation, and policy deployment.

---

📚 Found this guide helpful?

Explore more networking tutorials and insights at RJS Cloud Academy

Written by RJS Expert | Network Automation & Data Center Expert

Application Centric Infrastructure (ACI) Overview

Cisco ACI Complete Guide - Application Centric Infrastructure Deep Dive

🎯 What You'll Master:
Complete understanding of Cisco ACI architecture, from declarative policy models to spine-leaf topology, VXLAN overlay networks, and fabric initialization with real-world implementation

Table of Contents

1. What is Cisco ACI?
2. Nexus 9000 Switches - Foundation
3. ACI Network Topology
4. Cisco APIC Controller
5. Fabric Initialization
6. Spine-Leaf Architecture
7. IS-IS Infrastructure Routing
8. Endpoint Location & Policy
9. Physical, Virtual & Distributed
10. Important Points to Remember

What is Cisco ACI?

The Paradigm Shift

Cisco ACI represents a fundamental transformation in data center networking:

Traditional vs Modern Approach:

• Old: IP endpoint-based network → New: Application-based network
• Old: Manually configured network → New: Software-based network
• Old: Imperative configuration → New: Declarative model with Promise Theory

Understanding Promise Theory

What is Promise Theory?

• Instead of configuring every single port explicitly, you describe the desired application behavior
• ACI translates your intent from fabric-level policies down to hardware implementation
• You define what you want to accomplish, not how to accomplish it

🚖 Real-World Analogy:
Think of taking a taxi. You tell the driver your destination, not every turn, which route to take, or how fast to drive. Similarly, with ACI's Promise Theory:

• You declare: "Application A needs to communicate with Application B"
• ACI handles: VLAN assignments, routing, ACLs, QoS, and all underlying configuration

Core Architecture Principles

• Separation of Control Plane and Data Plane - Centralized policy management with distributed forwarding
• Holistic Architecture - Centralized automation with policy-driven application profiles
• Software Flexibility + Hardware Performance - Best of both worlds for dynamic workloads

Cisco Application Centric Infrastructure (Cisco ACI) in the data center is a holistic architecture with centralized automation and policy-driven application profiles. Cisco ACI delivers software flexibility with the scalability of hardware performance that provides a robust transport network for today's dynamic workloads.

This system-based approach simplifies, optimizes, and accelerates the entire application deployment lifecycle across data center, WAN, access, and cloud environments. This empowers IT to be more responsive to changing business and application needs, enhancing agility and adding business value.

Cisco ACI Key Characteristics

• Application-Centric Fabric Connectivity:
  • Multi-tier applications
  • Traditional applications
  • Virtualized applications
• Multivendor Support - Works with diverse hardware and software ecosystems
• Physical and Virtual Endpoints - Seamless integration of bare-metal and virtualized workloads
• Policy Abstraction - Simplify network configuration through high-level policies

ACI Starts with a Better Switch – Nexus 9000

The Cisco Nexus 9000 platform is the foundation of ACI, offering two distinct modes of operation.

Nexus 9000 Operating Modes

Mode 1: Standalone (NX-OS) Mode

Characteristics:

• Platforms: Nexus 9300 and 9500 series
• Behavior: Functions as a traditional Nexus L2/L3 switch
• OS: Enhanced version of NX-OS with automation capabilities
• Features: Best-in-class efficiency, low latency, high 10G/40G port density
• Use Case: Traditional switching with advanced programmability

Mode 2: ACI Mode

Characteristics:

• Platforms: Nexus 9300, Nexus 9500 switches
• Software: Runs "ACI version" of firmware
• Management: Managed by APIC (Application Policy Infrastructure Controller)
• Topology: Spine and Leaf fabric design
• Features: Application-centric representation, profile-based deployments, advanced automation
• Use Case: Modern data center with policy-driven networking

ACI Network Topology

ACI Topology is a CLOS Fabric

The ACI fabric follows a CLOS (non-blocking) topology design for optimal performance and scalability.

Fabric Design Rules

• All leafs uplink to all spines with 40/100 GigE connections
• APICs connect to leafs with redundant 10 GigE links
• Leafs do not plug into leafs - No horizontal connections
• Spines do not plug into spines - No horizontal connections
• Traffic flow pattern: Host → Leaf → Spine → Leaf → Host
• Scalability: Add more spines to scale out bandwidth

ACI Main Components

• Nexus 9K Spine Switches - Provide high-speed interconnection between all leaf switches
• Nexus 9K Leaf Switches - Connect to endpoints (servers, storage, services)
• Application Policy Infrastructure Controller (APIC) - Centralized policy and management controller

Cisco APIC (Application Policy Infrastructure Controller)

Cisco APIC is the brain of the ACI fabric - a policy controller that relays the intended state of policies to the fabric.

🔑 Critical Understanding:

• APIC is NOT in the data path - It's a management/policy plane controller
• APIC is NOT the control plane - Control plane is distributed across the fabric
• APIC holds the policy - It defines and pushes configuration to switches
• Fabric continues operating even if APIC is temporarily unavailable

APIC Key Features

• Policy Controller - Holds and distributes the defined policies
• Management Plane - Not in the control plane or traffic path
• Redundant Cluster - Three or more servers in highly available configuration
• Dual-Homed - Each APIC server connects to two leaf switches for resilience
• Scalability-Based Sizing - Cluster requirements determined by leaf port density (Verified Scalability Guide)
• Policy Instantiation - Translates high-level policies into switch configurations

APIC Hardware Platform

The Cisco APIC software runs on Cisco UCS C-Series server appliances with pre-installed software.

• Current Models (Two Generations):
  • Generation 2: APIC-L2 (Large) and APIC-M2 (Medium) - UCS C220 M4
  • Generation 1: APIC-L1 (Large) and APIC-M1 (Medium) - UCS C220 M3
• Management Interfaces:
  • GUI - Single pane of glass for entire topology (similar to UCS Manager)
  • CLI - Command-line interface for automation
  • APIs - RESTful APIs for programmatic access

ACI Fabric Initialization

ACI fabric supports automated discovery, boot, inventory, and system maintenance processes via the APIC.

Fabric Discovery and Addressing

The discovery process follows an automated sequence:

1. APIC finds a leaf - Initial connection established
2. Leaf finds the spines - Discovers all spine switches
3. Spines find all other leafs - Complete fabric topology mapped
4. Minimal GUI configuration - Simple setup steps required

Additional Initialization Functions

• Image Management - Centralized firmware distribution and upgrades
• Topology Validation - Verifies wiring diagram and performs system checks
• Automated Configuration - Self-configuring fabric with zero-touch provisioning

💡 Note: More detailed fabric initialization procedures covered in advanced sections

Spine-Leaf Topology

The spine-leaf topology makes the fabric easier to build, test, and support. Scalability is achieved by simply adding more nodes as needed.

Scalability Model

• Need more ports? Add more leaf nodes for connecting hosts
• Need more bandwidth? Add more spine nodes to increase fabric capacity
• Predictable growth - Linear scaling with deterministic performance

Spine-Leaf Advantages

🎯 Key Benefits:

• Simple and Consistent Topology - Predictable design pattern
• Scalability - For both connectivity (ports) and bandwidth (throughput)
• Symmetry - Optimized forwarding behavior across fabric
• Least-Cost Design - High bandwidth at minimal cost
• Low Latency - Maximum two hops for any host-to-host connection
• Low Oversubscription - Predictable bandwidth availability

Traffic Flow Characteristics

The symmetrical topology allows for optimized forwarding behavior:

• Any-to-Any Connectivity - All hosts can reach each other with same hop count
• Two-Hop Maximum - Host → Leaf → Spine → Leaf → Host
• Equal Cost Paths - Multiple paths available for load balancing
• No Spanning Tree - All links active simultaneously

IS-IS Fabric Infrastructure Routing

The fabric leverages a densely tuned IS-IS environment utilizing Level 1 connections within the topology for advertising loopback addresses.

IS-IS Role in ACI

Primary Responsibilities:

• Infrastructure Connectivity - Establishes routing between all fabric nodes
• VTEP Advertisement - Advertises VXLAN Tunnel Endpoint addresses (loopbacks)
• Multicast Trees - Computes multicast forwarding trees using FTAG (Forwarding TAG)
• Tunnel Announcement - Announces overlay tunnels from every leaf to all other fabric nodes

Technical Details

• Protocol Level: IS-IS Level 1 only
• Optimization: Tuned specifically for densely connected fabric environments
• VTEP Usage: Loopback addresses serve as VTEPs for integrated overlay
• Multicast FTAG: Vendor TLVs generate multicast forwarding tag trees

🔑 Key Understanding:
IS-IS in ACI is not used for routing application traffic. It's purely for:

• Infrastructure connectivity between switches
• Distributing VTEP addresses for VXLAN overlay
• Building multicast distribution trees

Application traffic uses the VXLAN overlay with distributed endpoint database.

Decoupling of Endpoint Location and Policy

The Cisco ACI fabric decouples the endpoint address from the location of that endpoint, defining endpoints by their locator or VTEP address.

How It Works

• Endpoints Identified - By IP or MAC address
• Endpoint Location - Specified by VTEP address (which leaf switch)
• Forwarding Mechanism - Occurs between VTEPs using VXLAN
• Transport Protocol - Enhanced VXLAN header format
• Reachability Database - Distributed database maps endpoints to VTEP locations

Benefits of Location-Policy Separation

Advantages:

• Mobility - Endpoints can move without policy changes
• Flexibility - Policy independent of physical location
• Scalability - Efficient use of network resources
• Simplification - Reduces configuration complexity

Physical, Virtual and Distributed

Multi-Hypervisor Support

Modern data centers require support for diverse workload types:

• Bare-Metal Servers - Physical servers directly connected
• Virtualized Workloads - VMs running on various hypervisors
• Containerized Applications - Modern microservices architectures
• Mixed Environments - Combination of all above

ACI Universal Support

ACI supports any type of endpoint with consistent policy application:

• Hypervisor Support:
  • VMware vSphere/ESXi
  • Microsoft Hyper-V
  • KVM (Kernel-based Virtual Machine)
  • Red Hat OpenStack
• Bare-Metal Servers - Direct server connectivity
• Containers - Kubernetes, Docker, OpenShift
• Unified Policy - Same policies apply regardless of endpoint type

Traffic Normalization

One of ACI's powerful features is encapsulation normalization:

How Normalization Works:

• Incoming Traffic: Can use different encapsulations:
  • Standard VLAN 802.1Q tags
  • VXLAN IDs
  • NVGRE (Network Virtualization using Generic Routing Encapsulation)
• ACI Normalization: Converts all traffic into Application Endpoint Groups (EPGs)
• Unified Treatment: ACI speaks any "language" and treats all endpoints with the same policy
• Benefit: Regardless of encapsulation type, consistent policy enforcement

ACI Behind the Scenes - Technical Deep Dive

⚠️ Under the Hood:

• Automated VXLAN Overlay - Tunnel system managed automatically
• Layer 2 and Layer 3 Gateways - Both supported for VXLAN
• VLANs with Local Significance - VLANs now have port-local meaning (not fabric-wide)
• IS-IS for Underlay - IS-IS routing protocol builds the transport network
• Leafs are VTEPs - Leaf switches serve as VXLAN Tunnel Endpoints
• VTEP to VTEP Transport - IP transport through spines connects all VTEPs

Important Points to Remember

Essential ACI Concepts

🎯 Critical Takeaways:

Topology & Protocols:

• No STP (Spanning Tree) - STP is not used because it blocks links; ACI uses all links simultaneously
• ECMP (Equal Cost Multi-Pathing) - Load balances traffic between leaf switches
• Layer 3 Fabric - ACI is fundamentally a Layer 3 fabric using IS-IS routing
• VXLAN Overlay - Used for building the overlay network on top of IP fabric
• LLDP Discovery - Protocol for discovering switches at Layer 2

Addressing & Assignment:

• Host-Based Networks - Every network in ACI is /32 (host-based routing)
• DHCP for Switch IPs - APIC uses DHCP for allocating IPs to each switch during discovery

Security Model:

• Whitelist Model - By default, everything is blocked unless explicitly allowed
• Security-First Approach - Excellent from a security perspective
• Explicit Contracts - Communication requires defined contracts between EPGs

Configuration & Management:

• Object-Based Storage - Everything configured is stored as objects and policies
• API Access - All configurations accessible using Cisco APIs
• XML/JSON Format - Configuration stored in standard formats
• API Configuration - Can be configured using RESTful APIs for automation

Key Takeaways & Next Steps

What You've Learned

• ACI Fundamentals - Promise Theory, declarative model, application-centric approach
• Nexus 9000 - Two modes (Standalone NX-OS and ACI), each with distinct use cases
• CLOS Topology - Spine-leaf architecture with predictable performance
• APIC Controller - Policy management plane (not data or control plane)
• Fabric Technology - IS-IS underlay, VXLAN overlay, distributed endpoint database
• Versatility - Support for physical, virtual, and containerized workloads

Design Principles Summary

Core Design Tenets:

• Simplicity: Consistent topology pattern
• Scalability: Add leafs for ports, spines for bandwidth
• Performance: Low latency, high bandwidth, low oversubscription
• Reliability: No single point of failure, APIC cluster redundancy
• Flexibility: Multi-hypervisor, multi-vendor, multi-encapsulation
• Security: Default-deny whitelist model with policy enforcement

What's Next?

Now that you understand the ACI architecture foundation, the next topics to explore include:

• Application Endpoint Groups (EPGs) - How applications are grouped and policies applied
• Contracts - How communication is permitted between EPGs
• Tenants - Multi-tenancy and resource isolation
• Bridge Domains & VRFs - Layer 2 and Layer 3 forwarding constructs
• Fabric Access Policies - Configuring physical connectivity
• Integration with External Networks - L3Out, L2Out configurations

🎯 Practice Challenge:
To solidify your understanding:

1. Draw an ACI fabric with 2 spines and 4 leafs - identify all connections
2. Trace a packet flow from Host A on Leaf1 to Host B on Leaf3
3. Identify what happens if one spine fails
4. Calculate bandwidth requirements for different oversubscription ratios
5. Design an APIC cluster for a 50-leaf fabric

Ready for the next level? Understanding ACI architecture is the foundation for implementing modern, automated data center networks. The next steps involve hands-on configuration of tenants, EPGs, contracts, and fabric policies to build production-ready ACI deployments.