this post was submitted on 25 Jul 2024

354 points (100.0% liked)

196

667 readers

38 users here now

Be sure to follow the rule before you head out.

Rule: You must post before you leave.

^other^ ^rules^

founded 1 year ago

MODERATORS

moss@lemmy.blahaj.zone

greembow@lemmy.blahaj.zone

funky_rodent@lemmy.blahaj.zone

PeachyMcPeachface@lemmy.blahaj.zone

remotelove@lemmy.ca

Roflmasterbigpimp@feddit.de

A_Very_Big_Fan@lemm.ee

qaz@lemmy.blahaj.zone

qaz@lemmy.dbzer0.com

qaz@lemmy.sdf.org

354

The Rule (lemmy.ml)

submitted 3 months ago by roon@lemmy.ml to c/196@lemmy.blahaj.zone

32 comments fedilink hide all child comments

you are viewing a single comment's thread
view the rest of the comments

[–] Sabata11792@ani.social 5 points 3 months ago* (last edited 3 months ago)

Some apps allow you to offload to GPU, and CPU while loading the active part of the model. I have a an old SSD that give me 500gb of "usable" ram set up as swap.

It is horrendously slow and pointless but you can do it. I got about 2 tokens in 10 minutes before I gave up on a 70b model on a 1080 ti.