Anyone playing around with local LLMs?
Finally got some downtime this weekend and set up a local RAG pipeline using Ollama and Llama-3. I just wanted to query my own messy folder of trading and financial PDFs and honestly it blew my mind how well it works. Running completely locally without paying any API fees to anyone
Idk why more people aren't doing this for personal stuff instead of paying for subscriptions. Anyone else building local AI tools for themselves? What's your stack looking like rn?
I tried running when Qwen 3.5B and 7B using my laptops 1650ti gpu a while back, felt really unusable. Nibba from nvidia recently claimed if we have clean data to train, 1B params are enough to have opus 4.7 level intelligence. If that's true, then local LLMs are the way forward. But as of now they feel useless to me, compared to free perplexity from airtel.
Ah yeah, a 1650ti might struggle a bit with context windows. Llama-3 8B runs surprisingly smooth on my current setup though. Fingers crossed those super smart 1B models drop soon!
