Interrupt Handling and Performance Tuning in Linux

Start Here

Get in touch with a
TriCore Solutions specialist

Blog | Mar 8, 2016

Interrupt Handling and Performance Tuning in Linux


Interrupt handling is one of the ways of enhancing system performance as per our needs and requirements because it is crucial for business to provide end users the best experience.

Introduction:

This blog will take you through the concept of interrupts, their handling by Linux and how we can use this handling for the most desired output that is -- tuning or enhancing the performance of an application or system. Once the system is in production we see some bottlenecks in the performance, sometimes it is hardware for example, memory, cables, NIC, etc. Wherein other times the culprit can be configured OS parameters, configuration, application parameters, etc. However, with continuous monitoring and analysing the situation we can work on eliminating significant bottlenecks or tune the system to improve the system performance. In this blog we will discuss one of such case, where we performed tuning as a way to interrupt the handling on the Linux systems for achieving better performance.

Linux_perform
Image source:  http://labs.sogeti.com/wp-content/uploads/2015/03/focus-and-performance1.png

What is an Interrupt?

Let me explain with the help of an example - Suppose a system is running a task but an urgent event occurs that needs the immediate attention of the CPU. What would happen in such a situation? Interrupts would be generated for urgent tasks.

We can say: 

An interrupt is an indication to the OS that an event has occurred that needs immediate attention.

In technical terms, an interrupt request (IRQ) is a request for service, sent at the hardware level. Interrupts can be sent by either a dedicated hardware line, or across a hardware bus as an information packet (a Message Signaled Interrupt, or MSI). For indication of an interrupt, a signal is sent to the processor called IRQ (Interrupt Request). When the kernel receives the IRQ, it calls the list of ISR (Interrupt Service Routine) associated with that interrupt, which performs all the actions required to handle that IRQ.

Interrupts have IRQ numbers or some standard acronyms/ abbreviations associated with them which we can see in the file: /proc/interrupts. For example:

[root@test~]# cat  /proc/interrupts

           CPU0       CPU1       CPU2       CPU3

 270:        379          0          0         85     Dynamic-irq  xenfb

271:          0          0          0          0     Dynamic-irq  xenkbd

272:    7612715     687736     400584     238329     Dynamic-irq  blkif

278:   27126385   28653240   16401236   18560993     Dynamic-irq  blkif

279:         25          0          0          0     Dynamic-irq  blkif

280:  888182639          0          0          0     Dynamic-irq  eth0

NMI:          0          0          0          0   Non-maskable interrupts

LOC:          0          0          0          0   Local timer interrupts

SPU:          0          0          0          0   Spurious interrupts

CAL:     258722     562089     889732     768631   Function call interrupts

Handling Interrupts in Linux:

Now that we know what an interrupt is, it is time to learn how to handle interrupts in Linux?

In the above output of /proc/interrupts, we have numbers in the first column as the IRQ number. Let’s say IRQ 280 had generated 645756827 interrupts for CPU0 but no interrupt for CPU1, CPU2 and CPU3. The CAL interrupt was distributed across all the CPUs, and the last two columns describe the nature of the interrupts - that is the interrupt type and the drivers registered to receive those interrupts. The last four interrupts: are represented by a standard acronyms like NMI, SPU etc.

For interrupt CAL, interrupts were not bound to only one CPU but distributed among many CPUs, which is the general or default handling of interrupts as mentioned in the /proc/irq/default_smp_affinity, where the value is represented in hexadecimal bit-mask.

[root@test~]# cat /proc/irq/default_smp_affinity

f

[root@test~]#

If we analyse, this default behaviour delay, we will see the processing each time the processor needs to build its instruction cache and at times it creates conflicts with the other processing occurring to the CPU. 

For our time-critical applications we can dedicate CPU for some specific types of interrupts. This helps increase the chances of having the data and instruction set for that interrupt in the cache of the processor which in turn speeds up the processing.

Next is how to dedicate CPU/ CPUs for single type of interrupt?

Each IRQ has its affinity or binding to the CPU which defines the CPU/ CPUs for that specific interrupt handling. This affinity is defined in file: /proc/irq/<IRQ num>/smp_affinity. The value stored in this file is a hexadecimal bit-mask representing all CPU cores in the system.

[root@test ~]#cat /proc/irq/280/smp_affinity

1

[root@test~]#

This forces the IRQ 280 to be served by CPU0.

Likewise, we can do the same for other IRQ numbers as well. 

By doing so, we can optimize the application and system’s performance.

Conclusion: Interrupt handling is one of the ways of enhancing system performance as per our needs and requirements because it is crucial for business to provide end users the best experience. This is especially true for the big database servers and the web servers. We can tune the interrupt affinity to the available CPUs considering the run time behaviour of system/ application and can avoid some performance issues and make the life easier for system admins.

For any questions on the topic click below:

Ask Mona