手把手教你在树莓派4B上运行LLaMA 7B模型

sudo su
apt-get update
apt-get upgrade

# 一些会用到的工具
apt-get install gcc g++ python3 python3-pip

# 安装python依赖
python3 -m pip install torch numpy sentencepiece

# build this repo
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
make

# obtain the original LLaMA model weights and place them in ./models
ls ./models
65B 30B 13B 7B tokenizer_checklist.chk tokenizer.model

# install Python dependencies
python3 -m pip install torch numpy sentencepiece

# convert the 7B model to ggml FP16 format
python3 convert-pth-to-ggml.py models/7B/ 1

# quantize the model to 4-bits
./quantize.sh 7B

./main -m ./models/7B/ggml-model-q4_0.bin -p "Hello world!" -t 8 -n 512

main: seed = 167882008main: seed = 167882008main: seed = 1678820083
llama_model_load: loading model from './models/7B/ggml-model-q4_0.bin' - please wait ...
llama_model_load: n_vocab = 32000
llama_model_load: n_ctx   = 512
llama_model_load: n_embd  = 4096
llama_model_load: n_mult  = 256
llama_model_load: n_head  = 32
llama_model_load: n_layer = 32
llama_model_load: n_rot   = 128
llama_model_load: f16     = 2
llama_model_load: n_ff    = 11008
llama_model_load: n_parts = 1
llama_model_load: ggml ctx size = 4529.34 MB
llama_model_load: memory_size =   512.00 MB, n_mem = 16384
llama_model_load: loading model part 1/1 from './models/7B/ggml-model-q4_0.bin'
llama_model_load: .................................... done
llama_model_load: model size =  4017.27 MB / num tensors = 291

system_info: n_threads = 4 / 4 | AVX = 0 | AVX2 = 0 | AVX512 = 0 | FMA = 0 | NEON = 1 | ARM_FMA = 1 | F16C = 0 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 0 | VSX = 0 | 

main: prompt: 'Hello world'
main: number of tokens in prompt = 3
     1 -> ''
 10994 -> 'Hello'
  3186 -> ' world'

sampling parameters: temp = 0.800000, top_k = 40, top_p = 0.950000, repeat_last_n = 64, repeat_penalty = 1.300000


Hello world! | Welcome to the new site.
We've been working hard on a few updates around here and it feels really great to finally share this

main: mem per token = 14368644 bytes
main:     load time = 113492.53 ms
main:   sample time =   987.31 ms
main:  predict time = 3881120.50 ms / 121285.02 ms per token
main:    total time = 4215260.00 ms

./main -m ./models/7B/ggml-model-q4_0.bin -p "Building a website can be done in 10 simple steps:" -t 8 -n 512

Building a website can be done in 10 simple steps:
Start with the end goal. Think about what your audience wants to achieve before they ever visit you site – and once there, how do these users interact on it? This is an important step as it will help formulate all of those great ideas into something that makes sense for their target marketplace!
2) Get Inspired & Develop a Mood Board: It's always good to have some kinda inspiration board. If you’re thinking about starting from scratch, use Pinterest or other sources (like Behance and Dribbble), as they will give your ideas a direction when it comes time for the visual design phase!
3) Strategize Your Content: Here's where things start to get tricky. You need some kinda content strategy in place here – do you have all of that information ready at hand? Now is also a good time as any to think about wireframes and how users will interact with your site once its built!
4) Do It With A Purpose: Are there particular goals for the user’s journey on this website, or simply trying to build an online presence? If so you may want to consider using marketing automation software like Hubspot that allows marketers and salespeople to work together better. This will help them organize your content into a more logical hierarchy – which is one of those things we love doing!
5) Visual Design: It's time for the fun part, where you think about colors (orange!) layout & typography, etcetera - and then start to design it using Illustrator or Photoshop. At this point in your project its important that all parties involved have a clear understanding of what needs to be done!
6) User Interface Design: This is when the UI designer comes into play – they should know how things interact with one another, understand user goals and create an experience so seamless it feels like second nature. The better this step goes down, you will end up having a much more engaging website at hand!
7) Coding Starts: After all of that planning is done here’s where the coding process begins – usually with HTML & CSS first and then followed by Javascript to make sure your site works flawlessly across multiple browsers.
8 ) Test, Test Again And Then Test Some More…: This phase can go on for a while (we know this from experience

main: mem per token = 14434244 bytes
main:     load time =  4570.61 ms
main:   sample time =   448.29 ms
main:  predict time = 138987.47 ms / 271.99 ms per token
main:    total time = 146495.30 ms