How to Connect APICs to the Cisco ACI Fabric
A comprehensive guide to Cisco ACI fabric initialization, APIC configuration, and fabric discovery process.
📋 ACI Fabric Setup Overview
To setup the Application Centric Infrastructure (ACI) Fabric, the following tasks need to be completed:
- ✅ Rack and Cable the Hardware
- ✅ Configure each Cisco APIC's Integrated Management Controller (CIMC)
- ✅ Check APIC firmware and software versions
- ✅ Check the image type (NX-OS/Cisco ACI) and software version of switches
- ✅ APIC1 initial setup
- ✅ Fabric discovery
- ✅ Setup the remainder of APIC Cluster
1. Rack and Cable the Hardware
APIC Connectivity
The APICs will be connected to Leaf switches. When using multiple APICs, we recommend connecting APICs to separate Leafs for redundancy purposes.
⚠️ Important Note for APIC M3/L3:
If it's APIC M3/L3, VIC 1445 has four ports (port-1, port-2, port-3, and port-4 from left to right):
- Port-1 and port-2 make a single pair corresponding to eth2-1 on the APIC
- Port-3 and port-4 make another pair corresponding to eth2-2 on the APIC
- Only a single connection is allowed for each pair
- All ports must be configured for the same speed, either 10G or 25G
Switch Connectivity
All Leaf switches will need to connect to spine switches and vice versa. This provides your fabric with a fully redundant switching fabric.
In addition to the fabric network connections, you'll also connect:
- 🔌 Redundant PSUs to separate power sources
- 🌐 Management Interface to your 1G out-of-band management network
- 💻 Console connection to a Terminal server (optional, but highly recommended)
2. Configure Cisco APIC Integrated Management Controller (CIMC)
When you first connect your CIMC connection marked with "mgmt." on the rear facing interface, it will be configured for DHCP by default. Cisco recommends that you assign a static address to avoid any loss of connectivity or changes to address leases.
Configuring CIMC via Console
You can modify the CIMC details by connecting a crash cart (physical monitor, USB keyboard and mouse) to the server and powering it on. During the boot sequence, it will prompt you to press "F8" to configure the CIMC.
NIC Mode Configuration
Dedicated Mode (Recommended):
- ✅ Utilizes the dedicated "mgmt." interface in the rear of the APIC appliance
- ✅ Separates CIMC platform management traffic
- ✅ Prevents fabric discovery issues
Shared LOM Mode (Not Recommended):
- ❌ Sends CIMC traffic over the LAN on Motherboard (LOM) port
- ❌ Shares bandwidth with APIC OS management traffic
- ❌ Can cause issues with fabric discovery if not properly configured
Aside from the IP address details, the rest of the options can be left alone unless there's a specific reason to modify them. Once a static address has been configured you will need to Save the settings & reboot.
Logging into the CIMC Web Interface
After a few minutes you should be able to reach the CIMC Web Interface using the newly assigned IP along with the default CIMC credentials:
• Username: admin
• Password: password
⚠️ It's recommended that you change the CIMC default admin password after first use.
To log into the CIMC, open a web browser to https://<CIMC_IP>. You'll need to ensure you have flash installed & permitted for the URL.
⚠️ Note: Launching the KVM console will require that you have Java version 1.6 or later installed. Depending on your client security settings, you may need to whitelist the CIMC address within your local Java settings for the KVM applet to load.
3. Check APIC Firmware and Software
All your APICs require to run the same version when joining a cluster. This may require manually upgrading/downgrading your APICs prior to joining them to the fabric.
📚 Reference: Instructions on upgrading standalone APICs using KVM vMedia can be found in the Cisco APIC Management, Installation, Upgrade, and Downgrade Guide for your respective version.
Switch nodes can be running any version of ACI switch image and can be upgraded/downgraded once joined to the fabric via firmware policy.
4. Check Image Type (NX-OS vs Cisco ACI)
For a Nexus 9000 series switch to be added to an ACI fabric, it needs to be running an ACI image. Switches that are ordered as "ACI Switches" will typically be shipped with an ACI image.
Converting NX-OS to ACI Mode
If you have existing standalone Nexus 9000 switches running traditional NX-OS, then you may need to install the appropriate image (For example, aci-n9000-dk9.14.0.1h.bin).
📚 Reference: See the Cisco Nexus 9000 Series NX-OS Software Upgrade and Downgrade Guide on CCO for your respective version of NX-OS.
5. APIC1 Initial Setup
Now that you have basic remote connectivity, you can complete the setup of your ACI fabric from any workstation with network access to the APIC. If the server is not powered on, do so now from the CIMC interface. The APIC will take 3-4 minutes to fully boot.
Next, open a console session via the CIMC KVM console. Assuming the APIC has completed the boot process, it should be sitting at a prompt "Press any key to continue…". Doing so will begin the setup utility.
Setup Dialogue Parameters
From here, the APIC will guide you through the initial setup dialogue. Carefully answer each question. Some of the items configured can't be changed after initial setup, so review your configuration before submitting it.
Configuration Parameters Explained
| Parameter | Description |
|---|---|
| Fabric Name | User defined, will be the logical friendly name of your fabric. |
| Fabric ID | Leave this ID as the default 1. |
| Number of Controllers | Set this to the number of APICs you plan to configure. This can be increased/decreased later. |
| Pod ID | The Pod ID to which this APIC is connected to. If this is your first APIC or you don't have more than a single Pod, this will always be 1. |
| Standby Controller | Beyond your active controllers (typically 3) you can designate additional APICs as standby. In the event you have an APIC failure, you can promote a standby to assume the identity of the failed APIC. |
| APIC-X | A special-use APIC model used for telemetry and other heavy ACI App purposes. For your initial setup this typically would not be applicable. (May be referenced as "ACI Services Engine" in future releases) |
Critical Configuration Parameters (Cannot Be Changed)
⚠️ TEP Pool (Tunnel Endpoint Pool)
This will be a subnet of addresses used for internal fabric communication. This subnet will NOT be exposed to your legacy network unless you're deploying the Cisco AVS or Cisco ACI Virtual Edge.
Recommendation: Assign an unused subnet of size between /16 and /21. Most customers allocate an unused /16 and move on. The size of the subnet used will impact the scale of your Pod.
⛔ Important: This value CANNOT be changed once configured. Having to modify this value requires a wipe of the fabric.
🚫 Subnet Restriction: The 172.17.0.0/16 subnet is NOT supported for the infra TEP pool due to a conflict with the docker0 interface. If you must use this subnet, you must manually configure the docker0 IP address to be in a different address space in each Cisco APIC before clustering.
⚠️ Infra VLAN
This is the VLAN ID for all fabric connectivity. This VLAN ID should be allocated solely to ACI, and not used by any other legacy device in your network.
Though this VLAN is used for fabric communication, there are certain instances where this VLAN ID may need to be extended outside of the fabric such as the deployment of the Cisco AVS/AVE.
Recommendation: Cisco recommends a VLAN smaller than VLAN 3915 as being a safe option as it is not a reserved VLAN on Cisco DC platforms as of today.
⛔ Important: This value CANNOT be changed once configured. Having to modify this value requires a wipe of the fabric.
⚠️ BD Multicast Pool (GIPO)
Used for internal connectivity. We recommend leaving this as the default or assigning a unique range not used elsewhere in your infrastructure.
⛔ Important: This value CANNOT be changed once configured. Having to modify this value requires a wipe of the fabric.
Submitting the Configuration
Once the Setup Dialogue has been completed, it will allow you to review your entries before submitting. If you need to make any changes enter "y" otherwise enter "n" to apply the configuration.
After applying the configuration allow the APIC 4-5 minutes to fully bring all services online and initialize the REST login services before attempting to login through a web browser.
6. Fabric Discovery
With our first APIC fully configured, now we will login to the GUI and complete the discovery process for our switch nodes.
When logging in for the first time, you may have to accept the certificate warnings and/or add your APIC to the exception list.
Registering Fabric Nodes
Navigate to: Fabric tab → Inventory sub-tab → Fabric Membership folder
From this view you are presented with your registered fabric nodes. Click on the Nodes Pending Registration tab in the work pane and we should see our first Leaf switch waiting discovery.
⚠️ Note: This would be one of the Leaf switches where the APIC is directly connected to.
To register our first node:
- Click on the first row
- From the Actions menu (Tool Icon) select Register
Node Registration Details
The Register wizard will require some details including:
| Field | Description |
|---|---|
| Node ID | The unique identifier you wish to assign. Cannot be changed once assigned without decommissioning. Common Practice: • Leaf switches: 100+ • Spine switches: 200+ |
| Node Name | Hostname for the switch. Can be modified later. |
| RL TEP Pool | Reserved for Remote Leafs usage only. Doesn't apply to local fabric-connected Leaf switches. |
| Rack Name | Optional field for organizational purposes. |
This information is provided to the APIC via LLDP TLVs. If a switch was previously registered to another fabric without being erased, it would never appear as an unregistered node. It's important that all switches have been wiped clean prior to discovery.
Node Bootstrap Process
Once the registration details have been submitted, the entry for this leaf node will move from the Nodes Pending Registration tab to the Registered Nodes tab under Fabric Membership.
The node will take 3 to 4 minutes to complete the discovery, which includes the bootstrap process and bringing the switch to an "Active" state. During the process, you will notice a tunnel endpoint (TEP) address gets assigned from your Infra TEP pool (such as 10.0.0.0/16).
In-Depth: Fabric Discovery Process
📡 Discovery Process Steps
- LLDP Neighbor Discovery: Cisco APIC uses LLDP to discover a switch
- DHCP Request: After successful discovery, the switch sends a request for an IP address via DHCP
- TEP Address Allocation: Cisco APIC allocates an address from the DHCP pool. The switch uses this address as a TEP address
- Boot File Download: In the DHCP Offer packet, Cisco APIC passes the boot file information. The switch retrieves this file via HTTP GET to port 7777 of Cisco APIC
- Firmware Loading: The boot file HTTP GET 200 OK response contains the firmware that the switch will load
- IFM Establishment: Cisco APIC initiates the encrypted TCP session when the switch is listening on TCP port 12183 to establish the policy element Intra-Fabric Messaging (IFM)
🔐 IFM Security
Communication between the various nodes and processes in the Cisco ACI Fabric uses IFM (Intra-Fabric Messaging), and IFM uses SSL-encrypted TCP communication.
Each Cisco APIC and fabric node has 1024-bit SSL keys that are embedded in secure storage. The SSL certificates are signed by Cisco Manufacturing Certificate Authority (CMCA).
A fabric node is considered active when the Cisco APIC and the node can exchange heartbeats through the IFM process.
Node Status States
Node status may fluctuate between several states during the fabric registration process. The states are shown in the Fabric Node Vector table. The APIC CLI command to show the Fabric Node Vector table is acidiag fnvread.
| State | Description |
|---|---|
| Unknown | Node discovered but no Node ID policy configured |
| Undiscovered | Node ID configured but not yet discovered |
| Discovering | Node discovered but IP not yet assigned |
| Unsupported | Node is not a supported model |
| Disabled | Node has been decommissioned |
| Inactive | No IP connectivity |
| Active ✓ | Node is fully operational |
💡 Note: ACI uses inter-fabric messaging (IFM) packets to communicate between the different nodes or between leaf and spine. These IFM packets are typically TCP packets, which are secured by 1024-bit SSL encryption, and the keys used for encryption are stored on secure storage. These keys are signed by Cisco Manufacturing Certificate Authority (CMCA). Any issues with IFM process can prevent fabric nodes communicating and from joining the fabric.
Completing Node Registration
After the first Leaf has been discovered and moved to an Active state, it will then discover every Spine switch it's connected to. Go ahead and register each Spine switch in the same manner.
Since each Leaf Switch connects to every Spine switch, once the first Spine completes the discovery process, you should see all remaining Leaf switches pending registration. Go ahead with registering all remaining nodes and wait for all switches to transition to an Active state.
7. Setup Remainder of APIC Cluster
With all the switches online & active, our next step is to finish the APIC cluster configuration for the remaining nodes.
Navigate to: System → Controllers sub menu → Controllers Folder → apic1 → Clusters as Seen by this Node folder
From here you will see your single APIC along with other important details such as the Target Cluster Size and Current Cluster Size. Assuming you configured apic1 with a cluster size of 3, we'll have two more APICs to setup.
Adding APIC2 and APIC3
At this point we would want to now open the KVM console for APIC2 and begin running through the setup Dialogue just as we did for APIC1 previously.
⚠️ Critical Requirements for Additional APICs
When joining additional APICs to an existing cluster it's imperative that you configure:
- ✅ Same Fabric Name
- ✅ Same Infra VLAN
- ✅ Same TEP Pool
- ✅ Unique Controller ID (set to ID 2 for APIC2, ID 3 for APIC3)
You'll notice that you will not be prompted to configure Admin credentials. This is expected as they will be inherited from APIC1 once you join the cluster.
Allow APIC2 to fully boot and bring its services online. You can confirm everything was successfully configured as soon as you see the entry for APIC2 in the Active Controllers view.
During this time, it will also begin syncing with APIC1's configuration. Allow 4-5 minutes for this process to complete. During this time, you may see the State of the APICs transition back & forth between Fully Fit and Data Layer Synchronization in Progress.
Continue through the same process for APIC3, ensuring you assign the correct controller ID.
🎉 Fabric Discovery Complete!
This concludes the entire fabric discovery process. All your switches & controllers will now be in sync and under a single pane of management.
Your ACI fabric can be managed from any APIC IP. All APICs are active and maintain a consistent operational view of your fabric.
Complete IFM (Intra-Fabric Messaging) Process
After all processes are completed, the fabric is ready for production configuration. The complete IFM steps are:
🔄 IFM Process Steps
- Link Layer Discovery Protocol (LLDP) Neighbor Discovery
- Tunnel End Point (TEP) IP address assignment to the node via DHCP
- Node software upgraded if necessary
- ISIS adjacency mode
- Certification Validation
- Start of DME Process on switches
- Tunnel Setup (iVXLAN)
- Policy Element IFM Setup
Fabric Initialization Task Checklist
✅ Post-Setup Tasks
- ✅ Configure APIC1
- ✅ Add first Leaf to fabric
- ✅ Add all Spines to fabric
- ✅ Add remaining Leafs to fabric
- ✅ Add remaining APICs to fabric
- 🔧 Setup NTP
- 🔧 Configure OOB Management IP Pool
- 🔧 Configure Export Policies for Configuration and Tech Support Exports
- 🔧 Configure Firmware Policies (For Upgrades)
🚀 Your ACI Fabric is Now Ready!
You can now proceed with tenant configuration, EPG creation, and policy deployment.
📚 Found this guide helpful?
Explore more networking tutorials and insights at RJS Cloud Academy
Written by RJS Expert | Network Automation & Data Center Expert
No comments:
Post a Comment