Hi All,

Is anyone out there doing self hosted LLM Artificial Intelligence for infrastucture use?

What about cloud based AI LLM tools for infrastructure build out.

Like building and configuring systems.
Setting up virtual machines, lxd's, and virtual networks.

I've just spoke with a friend who is a software engineer, and he is saying its eat or be eaten in the software development world. The company he works for CEO is working with only 1/3 the current dev staff, and still making significant progress using AI LLM tools.

I work in infrastructure, and don't hear much about AI in that space yet.

---
■ Synchronet ■ Lunar Outpost BBS

Re: AI LLM Artificial intelligence infrastructure
By: bbsing to All on Sun Mar 08 2026 09:47 pm

Hi All,

Is anyone out there doing self hosted LLM Artificial Intelligence
for infrastucture use?

i dont have much experience at that stuff but i have a friend who
is being pressured to come up with something like that.

he's the head of IT at a large staffing company.
the people pressuring him dont know much about it either. they are hoping to have AI handle almost everything.

What about cloud based AI LLM tools for infrastructure build out.

Like building and configuring systems. Setting up virtual machines,
lxd's, and virtual networks.

yeah my buddy did mention that. they want him to come up with intergration
for all that.

I've just spoke with a friend who is a software engineer, and he is
saying its eat or be eaten in the software development world. The
company he works for CEO is working with only 1/3 the current

dude they are using ai for everything. people are using it for writing emails and doing posts on linkedin and everything.


--
"Before using Wildcat....This Company did not have a convenient way of
looking after some of the richest clients in the world...Now we do!"
---
■ Synchronet ■ ::: BBSES.info - free BBS services :::

Re: AI LLM Artificial intelligence infrastructure
By: MRO to bbsing on Mon Mar 09 2026 04:18:59

i dont have much experience at that stuff but i have a friend who
is being pressured to come up with something like that.

he's the head of IT at a large staffing company.
the people pressuring him dont know much about it either. they are hoping to have AI handle almost everything.

yeah my buddy did mention that. they want him to come up with intergration for all that.

I'm in infrastructure and I'm wanting to figure some of this stuff out. It looks like LLM systems are not going away, so using, creating, maintianing these type of systems might take me through the next 16 years of my working life.

Trying to get a system capable of running LLM models equals lots of cash, and I'm on the fence if I should get into it, so I'm wondering if anyone out here is doing any stuff on the self hosting side.

dude they are using ai for everything. people are using it for writing emails and doing posts on linkedin and everything.

email is so silly. AI (LLM) systems communicating with each other on behalf of people is lame. People already are having problems communicating with each other the more assistance, will only attrify that ability. I think the point of email was for people to communicate, not for systems to communicate. There are lots of protocols for systems to communicate.

---
■ Synchronet ■ Lunar Outpost BBS

On 19/03/2026 22:51, bbsing wrote:

I've just spoke with a friend who is a software engineer, and he is saying its eat or be eaten in the software development world. The company he works for CEO is working with only 1/3 the current dev staff, and still making significant progress using AI LLM tools.

It's mostly hype and a lot of failed businesses (those keen on letting developers go) are going to be realised sooner than later.

AI is a toolset in a developer's toolbox. LLMs are one of those tools. In that way, they are no more or less than a compiler, debugger, or IDE. Should we shun those things and go back to punchcards?

People need carpenters to build a wooden chair from scratch with quality craftsmanship; they'll still need software developers in the future to ensure software is built with the same concept of quality craftsmanship.

The other thing to bear in mind: no one wants or needs yet another Photoshop, there's plenty of alternatives. The idea that "anyone can make their own" is not only false, but would also be a productivity failure for a company to engage in, not a productivity boost. They should have focused on their business, not reinventing someone else's (unless they're genuinely going to compete.)

Then there's the liability issue. Say a company relies on Excel, but they decide to vibe their own - now if it gets math wrong, who is liable? The Post Office Horizon scandal in the UK is a pre-AI example of this kind of ownership of responsibility meets liability issue.
---
■ Synchronet ■ Diamond Mine Online BBS - bbs.dmine.net:24 - Fredericksburg, VA USA

Re: AI LLM Artificial intelligence infrastructure
By: bbsing to All on Sun Mar 08 2026 09:47 pm

Is anyone out there doing self hosted LLM Artificial Intelligence for infrastucture use?

Haha, a friend of mine tried it, even with some model that was supposed to be good at doing stuff for you, and it totally messed up his OS install.

He tried it several times, even.

I would definitely not trust LLMs to do any live systems managing. You might be able to utilize one to create network install images or a one-time setup that you can actively work on with it, but you certainly wouldn't want to let it just do its own thing. For someone that knows what they are doing, it would probably take much less time for them to do it themselves than to try to have an LLM do things for them. It might save the time of someone who is clueless and has never been a sysadmin at all, but for a pro it would be a huge waste of time to have to constantly correct the LLM every time it does something wrong.

Speaking of which, I'm an unemployed infrastructure sysadmin, if you should happen to see anything open ;).

---
■ Synchronet ■ TIRED of waiting 2 hours for a taco? GO TO TACOPRONTO.bbs.io

Re: AI LLM Artificial intelligence infrastructure
By: bbsing to MRO on Mon Mar 09 2026 09:23 am

Trying to get a system capable of running LLM models equals lots of cash, an I'm on the fence if I should get into it, so I'm wondering if anyone out her

I've got a system set up with ollama and 24gb vram. I can show you what a hosted model can do via chat or something. It's really not very good.
A cloud-hosted one might do better, but the risk is still extremely high of it doing things you don't want.

---
■ Synchronet ■ TIRED of waiting 2 hours for a taco? GO TO TACOPRONTO.bbs.io

Re: AI LLM Artificial intelligence infrastructure
By: phigan to bbsing on Tue Mar 10 2026 14:01:15

Hi phigan,

Re: AI LLM Artificial intelligence infrastructure
By: bbsing to MRO on Mon Mar 09 2026 09:23 am

Trying to get a system capable of running LLM models equals lots of cash, an I'm on the fence if I should get into it, so I'm wondering if anyone out her

I've got a system set up with ollama and 24gb vram. I can show you what a

What models are you liking, and what sizes are you getting good TPS?
What do you think is good TPS?
Have you tried Deepseek-v3?
vllm?

hosted model can do via chat or something. It's really not very good.
A cloud-hosted one might do better, but the risk is still extremely high of it doing things you don't want.

13 years ago messing things up was my concern with scripting aspects of sys admin work. I worked with a Jr admin, and he was really liking powershell, but at the time was new to it. My worry was scripts have a great way have doing things quickly but also screwing things up quickly. He was a great team member and his scripting use inspired me to adopt his ideas. He left for a different job closer to home. I miss working with him. Now most of my work is via scripting because I have too much to do and without scripts it takes too long. I really get to know individual systems better with full hands on vs scripts, and I like knowing each system, its behavior, but efficeincy demands quickery.

I've been contemplating building an system for LLM personal lab work on infrastructure, with the idea of discriptive text for total orchistration, including builds, configuration, and management on windows/linux/bsd type systems. I'm having a hell of a time getting costs down. 24vgb is expensive, about as expensive as 32vgb rtx 5090.

I've looked at so many builds with rtx 3060, rtx 4090, rtx 5090. 3060's are gone, 4090's are way overpriced, and if they burn up, are likely not going to be the card to replace a broken device. rtx 5080s are around but the older cards are just as good, but their way over MSRP. Its been pretty dang frustrating.

Back in 2000, I was facing a similar issue. I was priced out of hardware and OS due to cost, so I couldn't build a lab to learn on. Now I'm in the same place. I don't really want to spend $8K on a build that is obsolete in two years. I also don't want to miss out on all the hype/fun/understanding.

I was hoping distributive computing would have been the thing instead of giant LLM type shtuff.

---
■ Synchronet ■ Lunar Outpost BBS

Re: AI LLM Artificial intelligence infrastructure
By: bbsing to All on Sun Mar 08 2026 09:47 pm

I've just spoke with a friend who is a software engineer, and he is saying its eat or be eaten in the software development world. The company he works for CEO is working with only 1/3 the current dev staff, and still making significant progress using AI LLM tools.

I get contradictory signals in that regard.

I think the people asking to get AI integrated with workflows and selecting employees according to their suitability for AI work are typically management types rather than people doing the actual work. In the end you get AI pushed into both things that benefit from it and things that don't.

You can get some LLMs to write good boilerplate for your Terraform/OpenTofu/Ansible/Whatever deployments. However, at this point you would be crazy to let an AI agent act as an orchrestrator for you.


--
gopher://gopher.richardfalken.com/1/richardfalken

---
■ Synchronet ■ Palantir BBS * palantirbbs.ddns.net * Pensacola, FL

Re: AI LLM Artificial intelligence infrastructure
By: bbsing to phigan on Tue Mar 10 2026 10:31 pm

What models are you liking, and what sizes are you getting good TPS?
What do you think is good TPS?
Have you tried Deepseek-v3?
vllm?

I'm not liking any of them. I haven't tried qwen2.5 out yet, I just downloaded it, but the rest of them aren't any good. At least for what I've been trying, which is code.

devstral:24b
codellama:34b
qwen3-coder:30b

I had the most success using 'Zed' as a front end to the qwen3-coder model. Codellama actually refused to write code at all. Devstral says it's going to write code but never does and gets stuck in a loop. I also tried installing claude and launching that with those same models. Codellama wouldn't open at all, saying it didn't support tools or whatever. Devstral again got stuck in a loop telling me it was going to do things but not actually doing them. qwen3-coder kept going back and forth on fixing one thing, breaking another, then breaking the first thing when it fixed the second thing, etc.

My GPU was second hand off Craigslist after scouring that and eBay quite a bit.

Still haven't found a good front end to use, but I have not tried Open WebUI just yet.

---
■ Synchronet ■ TIRED of waiting 2 hours for a taco? GO TO TACOPRONTO.bbs.io

Re: AI LLM Artificial intelligence infrastructure
By: Arelor to bbsing on Wed Mar 11 2026 14:23:49

I think the people asking to get AI integrated with workflows and selecting employees according to their suitability for AI work are typically management types rather than people doing the actual work. In the end you get AI pushed into both things that benefit from it and things that don't.

Management's dream is have no employees.

In the mind of those who don't do the work, it really seems these tools can do whatever anyone asks. They don't think about the quality and accuracy as a significant differentiator for sucess with competitors in the market. A lot of time management is focused on speed and efficency for protoctivity/cost reduction. Make widgets faster = more money. When mamagement recieves negative feedback from the markets, its usually too late for the employees they've let go.

You can get some LLMs to write good boilerplate for your Terraform/OpenTofu/Ansible/Whatever deployments. However, at this point you would be crazy to let an AI agent act as an orchrestrator for you.

I used to hear a lot about infrastructure as code, not as much over the past 3 years.

Probablistic determinates are a problem that doesn't seem to be going away.


But the media and hype machine is saying all jobs done on a screen will be targets for AI/LLMs to take over.

Whats a person to do?
Learn these new tools?
Stick with traditional methods?
A bit of both?

Cost is such an issue these days in the LLM space.

---
■ Synchronet ■ Lunar Outpost BBS