How can I do my research as a GPU poor?
I'd say the first step is to rule out LoRA. If LoRA is an option, it buys you more than just the rig savings:
- You can deploy multiple specialized LoRAs for different tasks
- It massively reduces your train-test latency
- You get upstream LLM updates for "free," maybe you can even add the training to your CI
Is there a different angle on the research you can take that doesn't require training models of that size?
When choosing research problems it's important to not only follow what's interesting but also what's feasible given your resources. I frequently shelve research ideas because they're not feasible given my skills, time, data, resources etc
AWS, Azure and GCP have programs for Startups and for Researchers that give free credits for their respective platforms. Try applying for those programs.
They usually give between $1000 and $5000 worth of credits, and they may have other requirements like being enrolled in college, but you should check each of their respective programs to find out more.
I have a couple of M40 with 24 gb on a desktop computer, i had to tape the pci-e to "scale it down to 1x". It's ok for interference and play around with small trainings, but I can barely infer a quantized version of 70b parameters. Training anything bigger than 3b parameters in human timescale is impossible. Either you scale it down or ask for a sponsor. It's frustrating because IT had always been approachable, until now...
Are you part of a university? Do they have resources or funding you can access?
Look into buying a used Dell c410x GPU chassis (they are $200 on my local CraigsList) and a Dell R720. Then add some cheap GPUs.
For a few thousand dollars you might get some processing power.
But you will never be able to pay your electric bill.
Would this help? https://github.com/evilsocket/cake
Do an internship at nvidiavdrover development?
Choose Your Weapon: Survival Strategies for Depressed AI Academics
Have been looking at this myself and it seems the advice isnt good.
Its basically Cost/Speed/Size Pick 2 or maybe even 1.
Some people have been able to run large LLM's on older slower CUDA gpus. A lot of them are truly ancient and have found their way back to ebay simply due to market conditions. They aren't good, but they work in a lot of use cases.
There are a couple of janky implementations to run on AMD instead. But reviews have been so mixed that I decided not to even test it. Ditto multi GPU setups. I thought that having access to 16 8g AMD cards from old mining rigs would have set me in good stead but apparently it benches roughly the same as just using a server with heaps of RAM because of the way the jobs are split up.
The cloud services seem to be the best option at the moment. But if you are going to spend 100 bucks vs going to spend 1000 bucks it might be worth just to fork out for the card.
Also honestly, hoping that someone else has a better idea in this thread because it will be useful to me too.