Recently mentioned elsewhere I was expanding my local GPU resources and wanted to briefly share my experience on the hot joplin topic du jour AI!
Had a 16gb RTX A2000 GPU previously in large selected cos I am very concerned about the huge power draw of Nvidia chipsets, and this has a maximum draw of 70w and idles much lower.
It was actually very good on RAG but there was underpowered for coding where you really need around 24gb to get the good stuff. But for me the power draw of the consumer cards that support that was unacceptable, so what I did was buy a 2nd 16GB A2000! This is not super efficient from a technology point of view, but with a combined 32Gb of VRAM i can run longer coding tasks and am ok to wait for the system to spit out results.
However, perhaps the biggest advantage is I now run my dual monitors off the CPU onboard graphics which actually works fine. That frees up the entire VRAM on card 1 and I'm getting much better performance with models that combined with embedding previously broke the 16gb barrier. Or perhaps rather Ollama was reserving some ram margin of error, because I was getting cpu/ram offloading with model combos that should have fitted in VRAM.
Whatever, the precise explanation a game changer for me and one that was not at all obvious having used reasonably high end Nvidia cards since the great financial crisis!
For RAG queries confined to 1 card, TDP jumps to the rated 65-70w on processing, while card 2 goes to about 15w for a few seconds then falls back to idle around 3-5w. When both cards are idle total GPU power draw is around 6 or 7W, I am delighted!