Internet in a box

[JTH]

I've been going down numerous rabbit holes, testing Llama AI models, running it in a VM, I'm losing about 10-15% overhead on resources, but we are still in beta and I don't want to alter the HOST system.

It started with the idea I wanted an Internet in a box type of Cyberdeck (fully offline devise) for when the world ends.
You can download all the WIKI, and of course all other sorts of documents you deem relevant to rebuilding the world.
I can then build an extensive library, using purpose built AI models to sift through it when I need quick indexing or answers.

If you have the right data and models, you can build advisors, like a Doctor, Farmer, etc

Anyhow, I rebuilt the Laptop, due to needing one large partition to store models and databases in one space. An 8-core AMD Ryzen 7 4700U with 16GB of ram, and 500GB storage (without a dedicated GPU), can run Small" Llama 3 8B Instruct models if they are heavily quantized. This is about the equivalent of Chat GPT-3.5 but with about half the context memory, so I can run similar (but shorter) task.

 

[nasa1974]

I've been going down numerous rabbit holes, testing Llama AI models, running it in a VM, I'm losing about 10-15% overhead on resources, but we are still in beta and I don't want to alter the HOST system.

It started with the idea I wanted an Internet in a box type of Cyberdeck (fully offline devise) for when the world ends.
You can download all the WIKI, and of course all other sorts of documents you deem relevant to rebuilding the world.
I can then build an extensive library, using purpose built AI models to sift through it when I need quick indexing or answers.

If you have the right data and models, you can build advisors, like a Doctor, Farmer, etc

Anyhow, I rebuilt the Laptop, due to needing one large partition to store models and databases in one space. An 8-core AMD Ryzen 7 4700U with 16GB of ram, and 500GB storage (without a dedicated GPU), can run Small" Llama 3 8B Instruct models if they are heavily quantized. This is about the equivalent of Chat GPT-3.5 but with about half the context memory, so I can run similar (but shorter) task.

Interesting. Good luck.

 

[TommyIV]

Cool stuff JTH,

So, which models have stood out to you so far? And what is the largest one you've been able to run on your VM?

When I find time this week, I'm thinking of downloading and playing with Qwen3-4B-Thinking-2507 off of LM studios.

 

[jonfresno]

You retired guys get into all sorts of things...

 

[JTH]

Cool stuff JTH,

So, which models have stood out to you so far? And what is the largest one you've been able to run on your VM?

When I find time this week, I'm thinking of downloading and playing with Qwen3-4B-Thinking-2507 off of LM studios.

Nothing stands out just yet, I've only been running basic reasoning test to compare how they answer the questions. Truth be told, it's both better than I expected (but slow), and also disappointing because I know I don't have the specs to run larger models. I'm slamming the CPUs hard which means when running the models I have no room for real-world task.

We both know what's going to happen here, I'm going to have build a machine...

 

[JTH]

Progress report.

I've moved the AI software unto the Host. Also managed to get my integrated graphics card to kick in some of the work on the smaller models. The Llama Server was easier, the Open Web UI has harder but it has a lot of options.

There is a ton of features under the hood here, everything you wished GPT can do, this already has it, you can control many parameters, like logic/creative temp, context size and so much more. I have the Voice Module loaded, but need to configure this as it's own server service. Next I'll work on the project folders, so I can store specialized instruction sets. Then at some point expand out and feed it news (it's not connected to the internet) so some of these models are unaware of present circumstances.

 

[JTH]

For the internet in a box topic, I've developed a fascination with 12-Volt low-powered processors like the Intel N200, something that sips little power and can be re-charged easily. Something small that can be carried, and can be protected with a faraday pouch. Not sure if you've noticed (it's been 10 years since I've built a PC) it now seems cheaper to buy vs. build. And frankly all of it is overpriced.

As for the AI project, the laptop can run up to 9B-Q5-KM models with no IGPU at about 4 words per sec (Super slow) I just don't have enough juice to run these at a fast pace. Smaller models run faster, but they are dumber like a 3rd grader. I did get internet searches to work but seem to be having issues with them parsing internet pages, they love to give you links, but they don't want to read the data. I broke all of it today trying to add Ollama to my Llama for the Open WebUI. So I'm going to remove it all and rebuild from scratch (again).

Also I put Linux on my Arm-Based ChromeOS tablet, cool right?
It is until you find out your USB-C Video output to external monitor no longer works... nor does your Netflix because they haven't developed the DRM (Digital Rights Mgmt). So yes, I'm the proud owner of a linux ARM tablet, but I feel like I only have about 60% of the linux features my X86-64 MX Linux has.

 

[tsptalk]

I asked grok to dumb down JTH's last post for me:

Hey kiddo! Imagine a grown-up guy who's super into building his own cool computer toys. He's writing a note to a friend about two fun (but kinda frustrating) projects he's messing with. I'll explain it like a bedtime story, super simple:

The "Internet in a Box" Idea

He's dreaming of making a magic little box that has the whole internet inside it — like a backpack-sized computer you can grab and take anywhere (even if the power goes out or something bad happens). He’s really excited about tiny, battery-friendly computer brains (like one called the Intel N200). These brains are awesome because they sip power like a tiny sip of juice — not a big gulp. You can recharge them easy-peasy with a car battery or solar charger. The whole thing is small enough to carry in your pocket almost, and he wants to tuck it into a special shiny bag called a "Faraday pouch." That bag is like an invisible superhero shield that blocks zappy radio waves so no one can spy on it or fry it with lightning or weird storms.

He also grumbles that back when he built big computers 10 years ago, it was cheaper to make your own. Now? It’s actually cheaper (and easier) to just buy one ready-made… but he thinks everything is way too expensive anyway. “Boo, prices!”

His AI Robot Brain Project

Next, he’s trying to build his own smart robot friend (called AI) right on his laptop.

• He can make the AI “think” using medium-big robot brains (the 9-billion-parameter kind). But it’s slooooow — only spits out about 4 words every second! That’s like waiting forever for your friend to finish a sentence.
• His laptop doesn’t have enough “juice” (power) to make it run fast.
• Tiny robot brains run quicker… but they’re super dumb, like a silly third-grader who doesn’t know much.
• He got the AI to search the real internet, which is cool… but it’s being lazy. It just hands you links (“go read this yourself!”) instead of actually reading the page and telling you the good stuff.

Then today he tried adding some extra magic software (stuff called Ollama, Llama, and Open WebUI — basically ways to talk to the AI easier). Oops! He broke EVERYTHING. So now he has to delete it all and start over from scratch… again. Classic “I touched it and it exploded” moment.

The Cool Tablet Adventure

Last part: He took his tablet (the kind that normally runs Google’s ChromeOS and has an ARM brain — that’s just a different family of computer parts) and put real Linux on it instead. Linux is a free, super-powerful computer operating system.

He says, “Cool, right?!” …until he found out two things broke:

• He can’t plug it into a big TV or monitor anymore with the USB-C cord (video stopped working).
• Netflix won’t play because it needs special “movie locks” (called DRM) that the new setup doesn’t have yet.

So now he’s the proud owner of a Linux tablet… but it only feels like 60% as good as his normal powerful Linux computer (the one with the x86-64 brain). It’s like getting a shiny new bike but two of the gears don’t work.

That’s the whole message! He’s having a blast tinkering but also hitting a bunch of “why won’t you work?!” moments. Grown-ups who love computers call this “fun frustration.”

 

[JTH]

I asked grok to dumb down JTH's last post for me:

Yes, that about sums it up, now I feel brave enough to set the clock on my VCR