scripts: add script to compare logprobs of llama.cpp against other frameworks

[ngxson]

Compare logits between llama.cpp and another inference engine using OpenAI-compatible server endpoints. Unlike compare-logits.py, it allows dumping logits from a hosted API endpoint. Useful when it's not possible to run both models locally. Example usage: Step 1: Dump logits from two different servers python scripts/compare-logprobs.py dump logits_llama.log http://localhost:8080/v1/completions python scripts/compare-logprobs.py dump logits_other.log http://other-engine:8000/v1/completions (optionally, you can add --api-key <key> if the endpoint requires authentication) Step 2: Compare the dumped logits python scripts/compare-logprobs.py compare logits_llama.log logits_other.log report.md 

[ngxson]

Just note that I'm no longer use the echo option because the support is hit-or-missed across frameworks

[pwilkin]

If this is a going to be a general tool I'd drop the prompt fragment about the tool call syntax, especially since we're not providing any tools.

[ngxson]

Tools and chat template are not handled here, we are using the raw completions endpoints.

The goal is a bit like perplexity test, where we don't actually care about the input text, we only care about the logits of the next predicted token

[ggerganov]

Tangentially, thinking if we can prepare a script that automates the comparison for a set of most prominent models:

1. Download latest HF repo of original model
2. Convert to GGUF BF16
3. Compare logprobs, generate report
4. Delete models
5. Goto 1. for next model

The main issue is to get hardware to run this periodically.

---

One more thing - this roughly tests the logprobs at mostly empty context. It would be useful to be able to run the comparison at larger depth - f.ex. logprobs for 100 tokens at 32k tokens depth. The reason is because some models change their parameters at larger depths and we should be able to test that too.

[pwilkin]

@ggerganov The scripts by @danbev in examples/model-conversion already do a lot of that. I routinely run causal-verify-logits.

As for the second task, I've been thinking of the same thing recently, in fact, I've already started working on a branch for that as well. Since the llama-logits tool is way too restricted for this, I've employed llama-cli (or the new llama-completion) and made it so that if you run it with --verbose and --n-predict 1, it dumps logits like llama-logits does. I've also refactored the run-org-model.py script to run long queries for the original transformer in batched chunks similar to the llama-cli behavior.

The biggest issue though is after processing 20k+ tokens the logits will almost never match exactly - the slight numerical differences will accumulate too much to make this test reasonable (I mean, even the fact that we're doing norm calcs in F16 in CUDA and Transformers does it in F32 will probably be enough). Unless we somehow make an exact copy of the KV cache and only process the tokens, I don't think we can reliably get a reasonable result from that.

[ngxson]

The main issue is to get hardware to run this periodically.

I think we could already somewhat automate this workflow by deploying everything to HF inference endpoint, which supports both vLLM and llama.cpp.

The main downside is that we can only do this with publicly available weights and it can be quite tricky to test a specific PR (since HFE only accepts publicly available docker image)

But still, I think we can already test some of the existing models (lmk which one you think we should test).

Will modify the script to pick N tokens at a deeper context length (unfortunately we cannot count exactly by tokens, because the runtime may not public the API for counting it)

[ggerganov]

@ngxson I'll see if I can make a setup on my DGX Spark - need to learn how to run vllm though.

@pwilkin There are various advantages of this comparison:

• Works through OAI API
• Can be used to verify against vllm which is what most model providers develop against

The biggest issue though is after processing 20k+ tokens the logits will almost never match exactly

The logprobs actually align quite well at long contexts since the result for token N is no longer a function of the result for token N-1 and hence there is no accumulation of numerical differences.