redhat_network_performance_tuning.pdf

FortuneXu

29页

2次

2025-09-13

免费下载

Red Hat Enterprise Linux Network Performance Tuning

Guide

Authors: Jamie Bainbridge and Jon Maxwell

Reviewer: Noah Davids

Editors: Dayle Parker and Chris Negus

03/25/2015

Tuning a network interface card (NIC) for optimum throughput and latency is a complex process

with many factors to consider.

These factors include capabilities of the network interface, driver features and options, the system

hardware that Red Hat Enterprise Linux is installed on, CPU-to-memory architecture, amount of

CPU cores, the version of the Red Hat Enterprise Linux kernel which implies the driver version,

not to mention the workload the network interface has to handle, and which factors (speed or

latency) are most important to that workload.

There is no generic configuration that can be broadly applied to every system, as the above

factors are always different.

The aim of this document is not to provide specific tuning information, but to introduce the reader

to the process of packet reception within the Linux kernel, then to demonstrate available tuning

methods which can be applied to a given system.

PACKET RECEPTION IN THE LINUX KERNEL

The NIC ring buffer

Receive ring buffers are shared between the device driver and NIC. The card assigns a transmit

(TX) and receive (RX) ring buffer. As the name implies, the ring buffer is a circular buffer where an

overflow simply overwrites existing data. It should be noted that there are two ways to move data

from the NIC to the kernel, hardware interrupts and software interrupts, also called SoftIRQs.

The RX ring buffer is used to store incoming packets until they can be processed by the device

driver. The device driver drains the RX ring, typically via SoftIRQs, which puts the incoming

packets into a kernel data structure called an sk_buff or “skb” to begin its journey through the

kernel and up to the application which owns the relevant socket. The TX ring buffer is used to

hold outgoing packets which are destined for the wire.

These ring buffers reside at the bottom of the stack and are a crucial point at which packet drop

can occur, which in turn will adversely affect network performance.

Interrupts and Interrupt Handlers

Interrupts from the hardware are known as “top-half” interrupts. When a NIC receives incoming

data, it copies the data into kernel buffers using DMA. The NIC notifies the kernel of this data by

Red Hat Enterprise Linux Network Performance Tuning Guide | Bainbridge, Maxwell 1

raising a hard interrupt. These interrupts are processed by interrupt handlers which do minimal

work, as they have already interrupted another task and cannot be interrupted themselves. Hard

interrupts can be expensive in terms of CPU usage, especially when holding kernel locks.

The hard interrupt handler then leaves the majority of packet reception to a software interrupt, or

SoftIRQ, process which can be scheduled more fairly.

Hard interrupts can be seen in /proc/interrupts where each queue has an interrupt vector in

the 1

column assigned to it. These are initialized when the system boots or when the NIC device

driver module is loaded. Each RX and TX queue is assigned a unique vector, which informs the

interrupt handler as to which NIC/queue the interrupt is coming from. The columns represent the

number of incoming interrupts as a counter value:

# egrep “CPU0|eth2” /proc/interrupts

CPU0 CPU1 CPU2 CPU3 CPU4 CPU5

105: 141606 0 0 0 0 0 IR-PCI-MSI-edge eth2-rx-0

106: 0 141091 0 0 0 0 IR-PCI-MSI-edge eth2-rx-1

107: 2 0 163785 0 0 0 IR-PCI-MSI-edge eth2-rx-2

108: 3 0 0 194370 0 0 IR-PCI-MSI-edge eth2-rx-3

109: 0 0 0 0 0 0 IR-PCI-MSI-edge eth2-tx

SoftIRQs

Also known as “bottom-half” interrupts, software interrupt requests (SoftIRQs) are kernel routines

which are scheduled to run at a time when other tasks will not be interrupted. The SoftIRQ's

purpose is to drain the network adapter receive ring buffers. These routines run in the form of

ksoftirqd/cpu-number processes and call driver-specific code functions. They can be seen

in process monitoring tools such as ps and top.

The following call stack, read from the bottom up, is an example of a SoftIRQ polling a Mellanox

card. The functions marked [mlx4_en] are the Mellanox polling routines in the mlx4_en.ko

driver kernel module, called by the kernel's generic polling routines such as net_rx_action.

After moving from the driver to the kernel, the traffic being received will then move up to the

socket, ready for the application to consume:

mlx4_en_complete_rx_desc [mlx4_en]

mlx4_en_process_rx_cq [mlx4_en]

mlx4_en_poll_rx_cq [mlx4_en]

net_rx_action

__do_softirq

run_ksoftirqd

smpboot_thread_fn

kthread

kernel_thread_starter

1 lock held by ksoftirqd

Red Hat Enterprise Linux Network Performance Tuning Guide | Bainbridge, Maxwell 2

of 29

免费下载

linux

关注

评论