Creating a Local LLM with Llama 3 & Oobabooga - A Beginner's Guide to AI

The post 30-second summary

Recently, Meta (formerly Facebook) announced that the Llama 3, Microsoft's (MS) Phi-3, etc. Open-source generative AI models take center stageI'm getting it. But these models rely on an internet connection to work, so what if you don't have internet access?

Obviously, you can't. However, in this post, we'll show you how you can. A solution to use AI systems in local environments using an open source program called Oobabooga and the latest AI model, Llama 3, which supports Korean.in this article.

Ubabuga Overview and Installation

Overview

Oobagooga is an open-source program that allows you to build AI models on your local computer and use AI systems without the internet. Through a simple user interface You can load and run interactive AI models, allowing you to All you need to know is which options to enable It's fairly accessible for beginners.

Oobagooga offers a wide range of settings options to customize the performance and output of your model (this is a bit tricky and will be covered in a separate post).

Install (macOS as primary)

So first, let's install Ubabuga on my computer by following the steps below.

1. Setup Files Download

Click 'Download' in the title above to download the file 'text-generation-webui-snapshot-2024-04-28.zip' to your Downloads folder. Right-click the file and click 'Open' to extract it.
* On Windows OS, you can find the 'start_windows.bat' file and Ubabuga will be installed in one step. At the end of the installation screen Below in the '5. Oobabooga threadCheck the local URL in the 'row' stepYou can.

2. Run the setup file

Press command (⌘) + spacebar and search for 'terminal' to launch a terminal windowand run the install command in the folder as shown below (it will automatically install various packages and configurations).

1Type cd command in the TP5T terminal to move a folder
cd downloads/text-generation-webui-snapshot-2024-04-28
Execute the # installation command.
sh start_macos.sh

3. Accessing virtual environments

The screen below shows the result of the second execution, and you can enter the virtual environment by executing 'conda activate ~else~/env' in the terminal.

# To activate this environment, use
# $ conda activate /Users/*******/downloads/text-generation-webui-snapshot-2024-04-28/installer_files/env 
# To deactivate an active environment, use
# $ conda deactivate                     

If you get a message like No module named 'yaml', run the pip install PyYAML command to install the packages.The first time I ran it, I had to install rich, accelerate, gradio, markdown, transformers, numba, and more with the pip install command. This is despite the fact that it's supposed to be a one-click install. (I'm sure there's another way to do this, and I'll update this in the future).

4. install requirements

Run the commands below to install the various packages required for macOS. Depending on your computer's specifications, it should take less than 5 minutes to install.

pip install -r "requirements_apple_silicon.txt"

5. Launch Oobabooga

Run the following command to start Oobabooga. To be precise, it is 'text generation web UI'. This will display the local URL in the terminal window as shown below, copy the URL and enter it into the address bar of your web browser to access it. Oobabooga will then run through your web browser as shown in the second image below (it will run its own server inside your computer).

./start_macos.sh

Download Llama 3 models

Click the Model tab at the top of the Ubabuga webUI, then paste the text below into each of the two fields above the Download button, and click the Download button to download the Llama3 Korean support model.

teddylee777/Llama-3-Open-Ko-8B-Instruct-preview-gguf
Llama-3-Open-Ko-8B-Instruct-preview-Q8_0.gguf

This model is based on the Junbeom Lee (beomi)You have pre-trained a model based on Meta-Llama-3-8B, Llama-3-Open-Ko-8B, on the Kyungrok Lee (teddylee777)created a model in GGUF format to run on low-end computers.

It will start downloading the model as shown below. The model is over 8GB, so it may take some time depending on your internet connection. Due to the woes of my construction apartment, it took over 10 minutes for the Lama 3 model file to download.

Llama 3 Oobabooga Load & Chatting

Now that you have downloaded the model, you need to make sure that Oobabooga is aware of this situation. Go to 1️⃣ and press the Refresh (?) button to recognize the downloaded Llama 3 model. 2️⃣ Press the dropdown button to select the Llama 3 model. 3️⃣ Click the Load button to mount the Lama 3 on Ooobabooga.

Click the 'Chat' menu at the top to chat with Lama 3, as shown below. I'm going to ask him about the 'King Sejong the Great Macbook Throwing Incident', and I'm curious to see what he'll say.

It's a verbal feast in English, not Korean, and I'm supposed to be able to show you something that works. I think my old Mac is a bit underpowered, so that might have something to do with it.

Let's take a look at the results of running it on a Windows OS-based gaming laptop with the specs below, but it's still responding in Korean, so I'm just making a big deal out of it. For now, we'll focus on building a local LLM for this post.

# Gaming Laptop with Windows OS Specifications
CPU: AMD Ryzen 7 5800H with Radeon Graphics 3.20 GHz
Graphics: AMD Radeon RX 6600M
Memory: 16.0 GB
OS: Windows 11 Home

Organize

That's it for how to build a local LLM, and you're now at least a Top 11 TP3T generative AI literate. Congratulations!

Installing Oobabooga took a little bit of focus, but other than that, I don't think there's much to it. I can now run Lama 3 without internet and use the Playing with AIis now possible.

Of course, the AI model's parameters are 8B, so the performance is not that good. If you have a server-class computer, I think you will be satisfied with 70B.

In the future, local LLM computers will be sold with specialized AI for each field, and you will be able to use AI services freely and securely, separated from the external internet network. Wouldn't that be one big business, or niche?

All content on 'secondlife.lol' is protected by copyright law. Unauthorized reprinting, copying, distribution, etc. is prohibited.