diff options
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/docs/customization.md | 24 |
1 files changed, 23 insertions, 1 deletions
diff --git a/docs/docs/customization.md b/docs/docs/customization.md index 096b42b2..74149ff9 100644 --- a/docs/docs/customization.md +++ b/docs/docs/customization.md @@ -14,7 +14,7 @@ Local Models - [Ollama](#run-llama-2-locally-with-ollama) - If you have a Mac, Ollama is the simplest way to run open-source models like Code Llama. - [GGML](#local-models-with-ggml) - Use llama-cpp-python to run a local server with any open-source model. -- [LlamaCpp](#llama-cpp) - Use llama.cpp directly instead of llama-cpp-python. +- [LlamaCpp](#llamacpp) - Use llama.cpp directly instead of llama-cpp-python. Open-Source Models (not local) @@ -117,6 +117,28 @@ config = ContinueConfig( ) ``` +### Llama.cpp + +Run the llama.cpp server binary to start the API server. If running on a remote server, be sure to set host to 0.0.0.0: +```shell +.\server.exe -c 4096 --host 0.0.0.0 -t 16 --mlock -m models\meta\llama\codellama-7b-instruct.Q8_0.gguf +``` + +After it's up and running, change `~/.continue/config.py` to look like this: + +```python +from continuedev.src.continuedev.libs.llm.ggml import GGML + +config = ContinueConfig( + ... + models=Models( + default=LlamaCpp( + max_context_length=4096, + server_url="http://localhost:8080") + ) +) +``` + ### Together The Together API is a cloud platform for running large AI models. You can sign up [here](https://api.together.xyz/signup), copy your API key on the initial welcome screen, and then hit the play button on any model from the [Together Models list](https://docs.together.ai/docs/models-inference). Change `~/.continue/config.py` to look like this: |