I think the LLM need about half the parameters in vram so a 30b model needs a video card with 16gb of vram. The 13b models need 8gb of vram.
Those aren't cheap.
this post was submitted on 17 Nov 2023
1 points (100.0% liked)
Homelab
22 readers
1 users here now
Rules
- Be Civil.
- Post about your homelab, discussion of your homelab, questions you may have, or general discussion about transition your skill from the homelab to the workplace.
- No memes or potato images.
- We love detailed homelab builds, especially network diagrams!
- Report any posts that you feel should be brought to our attention.
- Please no shitposting or blogspam.
- No Referral Linking.
- Keep piracy discussion off of this community
founded 1 year ago
MODERATORS