minute to read
jonasNo post last weekend? Oh no! I just couldn’t finish in time. Why? Because a few things happened, shifting this blog in a completely new direction: Project Skald.
You might have noticed from the blog post thumbnails that I’ve been using AI. But why do I use AI, and for what purpose? What do I even mean by “AI”? In keeping with the goal of this blog—to understand and over-engineer nerdy stuff—I decided to dive into AI, or more precisely, Large Language Models (LLMs).
A lot is happening in AI right now, and even if we only focus on LLMs (the “chat bots,” like ChatGPT), there’s no shortage of buzzwords. In Project Skald, I’ll do my best to explain them without overcomplicating things. My goal is to describe what I’m doing and (hopefully) take you along on that journey. I won’t post code or detailed tutorials; instead, I’ll share my ideas, the challenges I faced, and how my system works.
Why now? Why am I looking at AI when I’m also working on upgrading my cluster and introducing new services to BoniClaud?
Recently, a friend offered me an NVIDIA Jetson Nano—essentially a small computer, similar to a Raspberry Pi, but with one big difference: it has minimal CPU power but comes with an onboard GPU. Not just any GPU, either; it’s fully compatible with NVIDIA’s CUDA language and optimized for tensor operations. (Hopefully, I got that right!) In short, the Jetson is a mini computer tailored for developing and testing AI systems.
That got me thinking: How cool would a BoniClaud AI be? But also, what would it take to get something like that running, and what would it even do? Those questions kicked off this project, fueled by the realization that I didn’t really know how these AI systems work.
Project Skald will run alongside Project Svartalf. I still aim to post once a weekend, so these two projects will have updates on alternating weeks.
Skald focuses on LLMs—the kind of AI that completes text and can answer questions in a chat-like format. Project Skald won’t be part of BoniClaud because I don’t currently have the computing power to host such an AI in my cloud. (Unless I integrate something like the NVIDIA Jetson, but that wouldn’t be powerful enough for my use case.) Typically, a “lightweight” AI use case is image recognition, but I don’t really need that.
What I do need is an assistant for my notes. I play a lot of pen-and-paper RPGs, and those sessions generate a massive amount of text—about three gigabytes, which is roughly one million pages. The notes are often years old, and every now and then, someone asks, “Can anyone remember what happened at XYZ?” That’s why I keep notes. But the notes aren’t well organized, making them hard to search manually. This is exactly the sort of task a computer could handle—or better yet, a pen-and-paper LLM that can answer questions based on all those notes.
Project Skald will build a fully local RAG-enhanced LLM—that is, Retrieval Augmented Generation for a large language model.
What does that mean? Essentially, it’s a chatbot you can query in natural language, armed with the full knowledge of my notes.
That’s goal one. But I started this journey knowing almost nothing about LLM systems. I’ve used ChatGPT a lot and watched a few YouTube videos about how AI might take our jobs—and that’s about it. So, why not turn Project Skald into the beginner’s guide I wish I had? I’ll do my best to explain the terms and how things work, without diving into code. (Because, frankly, the code is shockingly uninteresting—it’s mostly plugging together existing frameworks and modules.)
While working on this project, I’ve also been thinking about when and how to use AI—and just as importantly, when not to use it. I might share more on this in the blog. For now, the biggest change is that I’m no longer using AI-generated thumbnails (at least for now). I’ll likely write a blog post on the reasons behind that choice, but in the meantime, I’ll be using other images as thumbnails.
As for the LLM aspect: yes, I’ll continue to have my blog posts proofread by AI because my grammar and spelling aren’t great. However, the AI doesn’t change the content; it only corrects typos and occasionally rephrases sentences. Unfortunately, I haven’t had much success generating satisfactory results with local LLM models—maybe that’s an optional goal for Project Skald to tackle later.
So, let’s build an AI! Well, more accurately, let’s configure a chatbot for my notes. But hey, that’s still AI, and I plan to over-engineer it. Could I just do a text search in my documents? Sure—but where’s the fun in that?