Hey everyone,
I’ve been lurking here for a while, but I finally ran into a performance bottleneck that has me scratching my head, so I figured it was time to reach out to the community.
I recently made the decision to migrate a portion of our high-throughput API traffic from a managed cloud environment to a dedicated bare-metal server. The goal was to reduce our monthly spend and, more importantly, to get as close to the hardware as possible to minimize tail latency. On a personal note, I’ve always been a bit of a hardware geek—there’s just something satisfying about hearing the fans spin up and knowing exactly what’s happening in the rack, rather than just clicking a button in a console.
The setup is pretty beefy, running on a newer PowerEdge rig. For the storage backend where we handle our heavy request logging and some local caching, I’m using a PERC H750 controller. I’ve got it configured with a mix of SAS drives for the critical logs and some high-capacity SATA drives for the less frequent backups. On paper, this HBA and Controller setup should be screaming fast, especially given the H750’s cache capabilities.
However, I’ve noticed a specific point of failure during peak traffic windows. When our request volume hits a certain threshold, the gateway starts throwing 504s, and I’m seeing massive I/O wait times in htop. It seems like the file-log plugin is essentially choking the system. Even with the H750 handling the heavy lifting for the RAID array, the interrupt requests seem to be conflicting with the way the gateway handles its worker processes. I’ve tried tweaking the mem_cache_size and playing around with the buffer settings, but the latency spikes persist whenever the controller is flushing its write cache to the SAS drives.
I’m starting to wonder if I’ve misconfigured the interaction between the OS-level disk scheduling and the hardware controller’s own logic. It’s frustrating because the CPU usage is sitting comfortably at 15%, but the throughput just tanks. I chose the PERC H750 specifically because I wanted that enterprise-grade stability for our SAS/SATA mix, but now I’m worried the “smart” features of the controller are actually introducing more jitter than a simpler HBA would have.
Has anyone else here moved back to bare metal for their gateway and run into these kinds of hardware-level resource contentions? Specifically, I’m curious if anyone has found a “sweet spot” for tuning the gateway’s internal buffer settings when dealing with high-performance RAID controllers that might be doing their own background housekeeping.
Is it possible that the gateway’s non-blocking I/O is somehow being “blocked” by the way the hardware controller handles disk writes, or am I likely looking at a more traditional Linux kernel tuning issue?