Using this set up we can even pause the insertion of metrics into the main cluster (in the event of a problem, or maintenance) with minimal impact to the user – alerts are processed separately so the user would only notice graphs no-longer updating. And when we switch it back on again we can process the backlog in just a few minutes with no data loss.



With the fixes in MongoDB 1.8, we have since enabled the balancing so these large collections are now spread out across the cluster. This helps with performance even more because inserts are spread across multiple machines.



And so we are now queuing MongoDB inserts using MongoDB itself! There are only a few threads doing inserts and this resulted in completely eliminating our queue spikes.