Anyone playing around with local LLMs?
Finally got some downtime this weekend and set up a local RAG pipeline using Ollama and Llama-3. I just wanted to query my own messy folder of trading and financial PDFs and honestly it blew my mind how well it works. Running completely locally without paying any API fees to anyone
Idk why more people aren't doing this for personal stuff instead of paying for subscriptions. Anyone else building local AI tools for themselves? What's your stack looking like rn?
Can you tell me about your local setup in short. I did try Google AI Edge gallery on my Mac, but nothing actually like model training in local, would certainly like to try out once. Can we train xgbm model in local (will search)?
Nah training requires way more compute.. are you that rich?
I get it now. And thanks for this post, it made be try it out. I tried gemma4 in local with ollama, using terminal commands; and then using Continue plugin in vs code editor. In vs code editor, I even compared the performance of gemma4 running in local on one side and gpt-4.1 on the other side (Continue plugin and Copilot plugin). But needless to say, gemma4 had to be handheld even for simple asks like “find the code which has created the xyz table”. So I think local models are still not capable enough for coding tasks. Or may be it is just gemma4 incompetency, I will later try some other models too, to see if any other models are usable for local coding work.
I tried running when Qwen 3.5B and 7B using my laptops 1650ti gpu a while back, felt really unusable. Nibba from nvidia recently claimed if we have clean data to train, 1B params are enough to have opus 4.7 level intelligence. If that's true, then local LLMs are the way forward. But as of now they feel useless to me, compared to free perplexity from airtel.
Ah yeah, a 1650ti might struggle a bit with context windows. Llama-3 8B runs surprisingly smooth on my current setup though. Fingers crossed those super smart 1B models drop soon!
