Interrupt handling is one of the ways of enhancing system performance as per our needs and requirements because it is crucial for business to provide end users the best experience.
This blog will take you through the concept of interrupts, their handling by Linux and how we can use this handling for the most desired output that is -- tuning or enhancing the performance of an application or system. Once the system is in production we see some bottlenecks in the performance, sometimes it is hardware for example, memory, cables, NIC, etc. Wherein other times the culprit can be configured OS parameters, configuration, application parameters, etc. However, with continuous monitoring and analysing the situation we can work on eliminating significant bottlenecks or tune the system to improve the system performance. In this blog we will discuss one of such case, where we performed tuning as a way to interrupt the handling on the Linux systems for achieving better performance.
Image source: http://labs.sogeti.com/wp-content/uploads/2015/03/focus-and-performance1.png
What is an Interrupt?
Let me explain with the help of an example - Suppose a system is running a task but an urgent event occurs that needs the immediate attention of the CPU. What would happen in such a situation? Interrupts would be generated for urgent tasks.
We can say:
An interrupt is an indication to the OS that an event has occurred that needs immediate attention.
In technical terms, an interrupt request (IRQ) is a request for service, sent at the hardware level. Interrupts can be sent by either a dedicated hardware line, or across a hardware bus as an information packet (a Message Signaled Interrupt, or MSI). For indication of an interrupt, a signal is sent to the processor called IRQ (Interrupt Request). When the kernel receives the IRQ, it calls the list of ISR (Interrupt Service Routine) associated with that interrupt, which performs all the actions required to handle that IRQ.
Interrupts have IRQ numbers or some standard acronyms/ abbreviations associated with them which we can see in the file: /proc/interrupts. For example:
[root@test~]# cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3
270: 379 0 0 85 Dynamic-irq xenfb
271: 0 0 0 0 Dynamic-irq xenkbd
272: 7612715 687736 400584 238329 Dynamic-irq blkif
278: 27126385 28653240 16401236 18560993 Dynamic-irq blkif
279: 25 0 0 0 Dynamic-irq blkif
280: 888182639 0 0 0 Dynamic-irq eth0
NMI: 0 0 0 0 Non-maskable interrupts
LOC: 0 0 0 0 Local timer interrupts
SPU: 0 0 0 0 Spurious interrupts
CAL: 258722 562089 889732 768631 Function call interrupts
Handling Interrupts in Linux:
Now that we know what an interrupt is, it is time to learn how to handle interrupts in Linux?
In the above output of /proc/interrupts, we have numbers in the first column as the IRQ number. Let’s say IRQ 280 had generated 645756827 interrupts for CPU0 but no interrupt for CPU1, CPU2 and CPU3. The CAL interrupt was distributed across all the CPUs, and the last two columns describe the nature of the interrupts - that is the interrupt type and the drivers registered to receive those interrupts. The last four interrupts: are represented by a standard acronyms like NMI, SPU etc.
For interrupt CAL, interrupts were not bound to only one CPU but distributed among many CPUs, which is the general or default handling of interrupts as mentioned in the /proc/irq/default_smp_affinity, where the value is represented in hexadecimal bit-mask.
[root@test~]# cat /proc/irq/default_smp_affinity
If we analyse, this default behaviour delay, we will see the processing each time the processor needs to build its instruction cache and at times it creates conflicts with the other processing occurring to the CPU.
For our time-critical applications we can dedicate CPU for some specific types of interrupts. This helps increase the chances of having the data and instruction set for that interrupt in the cache of the processor which in turn speeds up the processing.
Next is how to dedicate CPU/ CPUs for single type of interrupt?
Each IRQ has its affinity or binding to the CPU which defines the CPU/ CPUs for that specific interrupt handling. This affinity is defined in file: /proc/irq/<IRQ num>/smp_affinity. The value stored in this file is a hexadecimal bit-mask representing all CPU cores in the system.
[root@test ~]#cat /proc/irq/280/smp_affinity
This forces the IRQ 280 to be served by CPU0.
Likewise, we can do the same for other IRQ numbers as well.
By doing so, we can optimize the application and system’s performance.
Conclusion: Interrupt handling is one of the ways of enhancing system performance as per our needs and requirements because it is crucial for business to provide end users the best experience. This is especially true for the big database servers and the web servers. We can tune the interrupt affinity to the available CPUs considering the run time behaviour of system/ application and can avoid some performance issues and make the life easier for system admins.
For any questions on the topic click below: