LLama.cpp rust bindings.
The rust bindings are mostly based on https://github.com/go-skynet/go-llama.cpp/
Note: This repository uses git submodules to keep track of LLama.cpp.
Clone the repository locally:
git clone --recurse-submodules https://github.com/mdrokz/rust-llama.cppcargo build[dependencies] llama_cpp_rs = "0.3.0"use llama_cpp_rs::{ options::{ModelOptions,PredictOptions},LLama,};fnmain(){let model_options = ModelOptions::default();let llama = LLama::new("../wizard-vicuna-13B.ggmlv3.q4_0.bin".into(),&model_options,).unwrap();let predict_options = PredictOptions{token_callback:Some(Box::new(|token| {println!("token1:{}", token);true})), ..Default::default()}; llama .predict("what are the national animals of india".into(), predict_options,).unwrap();}The examples contain dockerfiles to run them
see examples
- Implement support for cublas,openBLAS & OpenCL #7
- Implement support for GPU (Metal)
- Add some test cases
- Support for fetching models through http & S3
- Sync with latest master & support GGUF
- Add some proper examples #7
MIT