Eval bug: wrong default threads in llama-bench

[GermanAizek]

Name and Version

latest commit: fa04659

llama-bench using by default wrong threads.

Linux server with NUMA nodes

Operating systems

Linux

GGML backends

CPU

Hardware

Supermicro X10dri Dual CPU

[o7si]

lstopo indicates a total of 36 physical cores (72 logical). The default n_threads value of 36 is likely intentional, aimed at maximizing performance by skipping SMT, rather than a calculation error.

llama.cpp/common/common.cpp

Lines 187 to 190 in 7f8ef50

	/**
	* Returns number of CPUs on system that are useful for math.
	*/
	int32_t cpu_get_num_math(){

[GermanAizek]

lstopo indicates a total of 36 physical cores (72 logical). The default n_threads value of 36 is likely intentional, aimed at maximizing performance by skipping SMT, rather than a calculation error.

llama.cpp/common/common.cpp

Lines 187 to 190 in 7f8ef50
/**

• Returns number of CPUs on system that are useful for math.
  */
  int32_t cpu_get_num_math(){

yes, but the problem is that 36 threads fewer tokens by default, and the system writes in monitoring that the cpu is util at 50%

isn't the task of the benchmark to test the system completely at 100%, I have fewer tokens if I don't set all CPU threads

[GermanAizek]

yes, but the problem is that 36 threads fewer tokens by default

devuan@devuan:~/GIT/llama.cpp/cmake-build-release/bin$ ./llama-bench -m Llama-3.2-1B-Instruct-Q2_K.gguf -p 512 -n 128
| model | size | params | backend | threads | test | t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | --------------: | -------------------: |
| llama 1B Q2_K - Medium | 546.50 MiB | 1.24 B | CPU | 36 | pp512 | 435.77 ± 3.03 |
| llama 1B Q2_K - Medium | 546.50 MiB | 1.24 B | CPU | 36 | tg128 | 24.68 ± 0.06 |

build: 8b11deea (6881)
devuan@devuan:~/GIT/llama.cpp/cmake-build-release/bin$ ./llama-bench -m Llama-3.2-1B-Instruct-Q2_K.gguf -p 512 -n 128 -t 72
| model | size | params | backend | threads | test | t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | --------------: | -------------------: |
| llama 1B Q2_K - Medium | 546.50 MiB | 1.24 B | CPU | 72 | pp512 | 480.15 ± 3.27 |
| llama 1B Q2_K - Medium | 546.50 MiB | 1.24 B | CPU | 72 | tg128 | 29.53 ± 0.14 |

build: 8b11deea (6881)

[taronaeo]

Using physical core count vs. logical core count should have been an intentional change and not a bug. SMT has been shown to degrade performance and recommended to be turned off in s390x's build guide here: https://github.com/ggml-org/llama.cpp/blob/master/docs/build-s390x.md#3-smt-vs-nosmt-simultaneous-multithreading

and the system writes in monitoring that the cpu is util at 50%

That's because your system uses SMT by default so total threads = physical core count x 2. Using 36 out of 72 available threads is exactly using 50% as reported by your monitoring solution.