MicrosoftHi, I'm trying to reproduce your results for "Time optimization" case, but have got different optimum
I use https://github.com/azure/AI-benchmarking-guide with following config: config-tp1-lat.json ,it use same parameters for input_output_sizes="1024,128" , but have got slightly different result, see batch_size_vs_perf_no_gp_ctx.png
 
In general graph pattern is the same, but batch_size optimum is different, optimum batch_size is ~80. I suspect that this is just a side effect of our environment is different. Can you please post which version of TensorRT-LLM you use and which parameters was used for engine-build and benchmark execution.
