Low-rank adaptation (LoRA) ... has some advantages over previous methods:
- It is faster and uses less memory, which means it can run on consumer hardware.
- The output is much smaller (megabytes, not gigabytes).
- You can combine multiple fine-tuned models together at runtime.
This is great news for my dream of building a fine-tuned interactive messenger, that can deliver a message on my behalf by training it on my personality & the information I want to convey.
Now just add text to speech and a talking head, as discussed in that other submission about cloning yourself with AI... https://news.ycombinator.com/item?id=35280418
> The weights for LLaMA have not yet been released publicly. To apply for access, fill out this Meta Research form.
Cute. ;)
LoRA has actually been around for a little while! I first saw it when it became popular in fine-tuning models quantized down to about 8 bits or so. I'm sure it's doing stuff in the 4bit range now! :D
I believe it's a core toolbox piece of tech required to really push the limits of LLMs either in original training or in inference. Similar sort of to how batch norm was for convolutional neural networks. I look forward to seeing how this will be applied in the future.
The easiest way to run alpaca Lora locally is with this little known fork [1] that uses Docker. You’ll be up and running in about 20 min with pretty much any modern consumer Nvidia GPU.
Hi All, I have a noob question. I have been reading about Alpaca and Alpaca Lora. I have a use case in which I want to fine tune/train Alpaca Lora on a large corpus of books which are in the txt format. I know for Alpaca, the data was in "Instruction : Prompt" format. however, my text is huge and is not in that format. It's simply a library of books and journal articles. I want to be able to ask a question and the model answers based on the books I trained it on. I also want to be able to ask general questions for example which books discussed topic x or y.
I have tried OpenAI's API to create embeddings, but I want to use Alpaca.
I really appreciate your help.
I love these idea of LoRAs for LLMs.
Has anybody made a llama/alpaca erebus model? I read about them in the oobabooga docs and a locally-run language model fine tuned on literotica could be the funniest thing I’ve ever seen.
I guess this LoRA is the missing piece.
NVIDIA stated recently that GPT bots will become one million times more powerful in ten years. Many people doubted that.
With LoRA, I see a much higher improvement. These guys claim a 10000 times reduction in parameter size. A different way to look at it, is that with the current hardware you can train a model that has 10000 times more parameters. If you add a 100x improvement in hardware in 10 years (not at all unrealistic), that's the million. But we will have significant improvements in training methods too.
Can a model be fine-tuned “online”?
If cost wasn’t an issue, could I fine-tune a model in real time, while also using it for inference?
So they use cog before installing it? Apparently this wasn’t proofread.
Also, is it just me or there are currently more ways to run LLMs on a CPU than on a GPU springing up on GitHub? I have hacked my own, but my chat UI is awful, so what is the nicest, pre-packaged CUDA-friendly way to run this now?
How does LoRA save more than 50% of the memory usage? I see that the weight updates have much lower memory footprint by virtue if being low rank. But you still need the dense weights for the forward pass dont you?
Anyone else click on this thinking it was about the wireless protocol?
It feels like I'm living in a cartoon with all these terms: > In this blog post, we’ll show you how to use LoRA to fine-tune LLaMA using Alpaca training data.
This looks fantastic. Will try replacing our current fine-tuned FLAN-UL2 model with this.
I wonder how the devtooling around this will evolve. Seems like a matter of days until someone creates a GUI wrapper around this, and obviates the need to use programmer time for fine-tuning