Use the cache-bin limit for tcache_nslots_small_max - #2978

Hey @xinydev , thanks for this, it looks like correct fix, but it may become obsolete after #2965 (assuming we go with it). If we do not we will merge this one in. We are trying to simplify option space and logic around tcache while potentially even improving fast path ratio.

No problem. I’m also really looking forward to the Adapt tcache fill and retention feature. Since we use a 64 KB page size by default, Adapt tcache fill is even more important for us. Thanks!

No problem. I’m also really looking forward to the Adapt tcache fill and retention feature. Since we use a 64 KB page size by default, Adapt tcache fill is even more important for us. Thanks!

It is on a dev branch now. So, try just setting your tcache per bin sizes the way you want it and the rest should be automatic. I am curious to see if algorithm works well for you. We wanted to simplify options space and significantly improve tcache hit ratio (reduce number of fills and flushes) while not significantly increasing memory. It looked better on our internal set of services, but is it a good trade-off in general it is not easy to say. That is why your data points will be valuable.

Sounds great. I’ll try it on some internal services next week.

We currently set ncache_max pretty high, up to 2048, to improve tcache reuse. But not every thread needs that much, and some may need more. Adapt-fill sounds like a really good fit here. I’ll test it and see how it works.

@xinydev It is pretty common to have unbalanced threads. Besides adaptive filling, another approach worthy trying is to reduce the global tcache_ncached_max and set busy threads' ncached_max to a higher value using the mallctl during thread initialization. In this way, the idle threads have fewer reserved space, which saves memory, while busy threads can use tcache more aggressively, effectively reusing the memory saved from idle threads. This can theoretically reach a net CPU win without memory regression. Yet more efforts are needed there because the ncachced_max tuning is per-thread and has a much larger search space compared to simply turning adaptive fill on.

If you have a chance to try either approach, would really appreciate any data points shared.

@spredolac @guangli-dai
I ran an A/B test on this. CPU and memory were basically flat, with no obvious change either way.

The tcache-related metrics looked like this:

• nmalloc / nrequests: 0.452532 → 0.487512, +7.73%
• ndalloc / nrequests: 0.449411 → 0.484967, +7.91%
• nfills / nrequests: 0.000830651 → 0.000551329, -33.63%
• nmalloc / nfills: 544.79 → 884.25, +62.31%
• nflushes / nrequests: 0.002248338 → 0.002339874, +4.07%
• nreslabs / nrequests: 0.001623812 → 0.001816971, +11.90%

So it does look like we’re doing fewer fills, while bringing in more objects per fill. nreslabs also went up quite a bit.

@spredolac @guangli-dai I ran an A/B test on this. CPU and memory were basically flat, with no obvious change either way.

The tcache-related metrics looked like this:

• nmalloc / nrequests: 0.452532 → 0.487512, +7.73%
• ndalloc / nrequests: 0.449411 → 0.484967, +7.91%
• nfills / nrequests: 0.000830651 → 0.000551329, -33.63%
• nmalloc / nfills: 544.79 → 884.25, +62.31%
• nflushes / nrequests: 0.002248338 → 0.002339874, +4.07%
• nreslabs / nrequests: 0.001623812 → 0.001816971, +11.90%

So it does look like we’re doing fewer fills, while bringing in more objects per fill. nreslabs also went up quite a bit.

@xinydev , thanks for the metrics. This is good news (that is all flat for you as well). I am surprised about fills going up, but based in the numbers calculated overall fills+flushes going down by 6.1%. The number we like especially is -33% in fills. This is so because we plan to do some work on slow path and running fewer slow-paths gives us better buffer. Given that all is flat I am not worried abut reslabs, this is a product of more objects being pulled in fills I think.