The MySQL “swap insanity” problem and the effects of the NUMA architecture

Jeremy Cole

Update: Also read A brief update on NUMA and MySQL.

The “swap insanity” problem, in brief

When running MySQL on a large system (e.g., 64GB RAM and dual quad core CPUs) with a large InnoDB buffer pool (e.g., 48GB), over time, Linux decides to swap out potentially large amounts of memory, despite appearing1 to be under no real memory pressure. Monitoring reveals that at no time is the system in actual need of more memory than it has available; and memory isn’t leaking, mysqld‘s RSS is normal and stable.

Normally a tiny bit of swap usage could be OK (we’re really concerned about activity—swaps in and out), but in many cases, “real” useful memory is being swapped: primarily parts of InnoDB’s buffer pool. When it’s needed once again, a big performance hit is taken to swap it back in, causing random delays in random queries. This…

View original post 3,035 more words

NUMA- linux tuning — this is humbling level of complexity

This is the processor i have on my laptop. I followed up with this hypothesis and ran numactl –hardware which gives me this output:

available: 1 nodes (0)
node 0 cpus: 0 1 2 3 4 5 6 7
node 0 size: 3971 MB
node 0 free: 634 MB
node distances:
node 0
0: 10

A view at cat /proc/cpuinfo gives me this

processor : 6
vendor_id : GenuineIntel
cpu family : 6
model : 58
model name : Intel(R) Core(TM) i7-3632QM CPU @ 2.20GHz
stepping : 9
microcode : 0x

OK, so looking up the processor points me to this Unfortunately that link mentions nothing about the actual architecture involved. Am left to guess that it’s because i have only 4G RAM (all allocated to one node) or my processor just does not belong in the category i thought it does.
Damn it. it’s time figure out this hypothesis. Time to look for other ways to test this.

This link suggests that mine does come under Ivy-bridge, but something called Ivy bridge(Mobile). Ivy bridge is the successor architecture after SandyBridge. That means i was right about the architecture (if only in terms of direction) and the next test is to add RAM and then see if numactl gives a better output. I have a feeling there might be more to this problem than meets the eye.(Like Linux installation time default ubuntu kernel does not have numa optimization or something like that .., but we’ll see. only one way to find out.)

This is a problem i have faced before, though on a different vps hosted server(64G RAM)

These are some other interesting links