Unlocking AI Magic: A Beginner’s Guide to Setting up and Running Ollama Locally

Ollama

Introduction

Artificial Intelligence has evolved rapidly, and now you can harness the power of large language models right on your local machine. In this guide, we’ll walk through the steps of setting up Ollama, a versatile tool, on your system. Ollama allows you to run models like Llama 2, Mistral, and more, giving you the flexibility to customize and create your own AI experiences.

Ollama opens the door to an array of possibilities by offering a local environment for running AI models. Whether you’re a developer, hobbyist, or simply curious about AI, having the ability to experiment with models like Llama 2 and other open source LLM can be a game-changer.

Getting Started

System Requirements

Before diving into the installation, ensure that your system meets the following requirements:

• Processor: Core i5 9300H or equivalent
• RAM: 16GB or Higher
• GPU: Nvidia GTX 1650 with 4GB VRAM (CUDA-enabled for optimal performance) or Higher
• Storage: SSD
• Operating System: Windows 10 / Windows 11 and Ubuntu WSL2 (Any distro with nvidia cuda support) or any other linux based system with CUDA support

Enabling WSL2 in your windows system

If you have wsl 1 installed on your machine then you will have to update it to wsl2. Also install the kernel package, I have mentioned the link below.

Check this link

https://learn.microsoft.com/en-us/windows/wsl/install

https://wslstorestorage.blob.core.windows.net/wslblob/wsl_update_x64.msi

Now, lets dive into Ollama 😃

Open your wsl by searching wsl in your windows search.

Ubuntu 22.04 wsl2

You can simply execute the following command to install Ollama in your system:

curl https://ollama.ai/install.sh | sh

This will basically install Ollama tool in your Ubuntu machine. Now we are ready to experiment with Ollama 😃

Now that Ollama is installed, let’s bring it up:

$ ollama serve

karthik@LAPTOP-ERPLBOBN:~/ollama$ ollama serve
2024/01/28 12:00:52 images.go:808: total blobs: 27
2024/01/28 12:00:52 images.go:815: total unused blobs removed: 0
2024/01/28 12:00:52 routes.go:930: Listening on 127.0.0.1:11434 (version 0.1.20)
2024/01/28 12:00:53 shim_ext_server.go:142: Dynamic LLM variants [cuda rocm]
2024/01/28 12:00:53 gpu.go:88: Detecting GPU type
2024/01/28 12:00:53 gpu.go:203: Searching for GPU management library libnvidia-ml.so
2024/01/28 12:00:53 gpu.go:248: Discovered GPU libraries: [/usr/lib/wsl/lib/libnvidia-ml.so.1]
2024/01/28 12:00:53 gpu.go:94: Nvidia GPU detected
2024/01/28 12:00:53 gpu.go:135: CUDA Compute Capability detected: 7.5

If you see output similar to the one provided, Ollama is up and running. Note the GPU detection details for optimal performance.

Lets open another wsl window

WSL new window

Keep watching the Ollama serve logs when we run an LLM model locally.

We will now install TinyLlama model. TinyLlama is a compact model with only 1.1B parameters. This compactness allows it to cater to a multitude of applications demanding a restricted computation and memory footprint.

$ ollama pull tinyllama

Run Tinyllama

$ ollama run tinyllama

Now, you can interact with Tinyllama, asking questions and exploring its capabilities.

karthik@LAPTOP-ERPLBOBN:~$ ollama run tinyllama
>>> Hi llama tell me about yourself
Sure, I'd be happy to share a little bit about myself! As you can see, I'm a friendly and approachable AI. Here are some of the things that make me unique:

1. I'm always available for help and support - whether it's with a simple task or a complex problem, I'm here to answer any questions or provide guidance. 2. I'm very smart and can learn from my
experiences - if I encounter a new situation, I'll try to figure out what's going on and come up with ways to overcome it. 3. I'm also curious and eager to learn - I love exploring new ideas and trying
out new things. 4. I have a natural sense of humor - I've been around for quite some time, so I can keep a joke going and make people laugh. 5. I'm always up-to-date with the latest technology and trends
- if there's something new and interesting happening in my field, you can bet I'll be among the first to hear about it! Overall, I try to approach everything I do with a positive and optimistic attitude,
even when things are tough or challenging. I believe that with hard work and persistence, anything is possible.

It is amazing right ?. That’s the magic of open source. 😏

You can also use Langchain to interact with tinyllama and build awesome applications!

I hope this tutorial helps you in setting up open source LLM in local machine. If you face any challenge or issue feel free to put comments.

Congratulations! You’ve successfully set up and run Ollama on your local machine. Dive into the world of AI, explore different models, and unleash your creativity. Experiment, learn, and enjoy the magic of AI right at your fingertips.

Explore more models on Ollama’s official library and join the growing community of AI enthusiasts. Happy coding!