There is a famous transformer model named T5 from Google, and also S4, S4 and S6 (Mamba) in the LLM space, so it is not unusual naming.

I propose T-POT (Tensor Product attentiOn Transformer)

If you don’t like the title, wait till you see this acronym:
„… we introduce the
Tensor ProducT ATTenTion Transformer (T6), a new model architecture…“

My kingdom for renaming this paper to something like &quot;Tensor Product Attention is a Memory-Efficient Approach for Long-Sequence Language Modeling&quot;

EA doesn&#x27;t quite fit in the same umbrella. EA has a constant cache size (it&#x27;s just another classical recurrent architecture inspired by approximating transformers), where this paper gives speedups to a variety of true attention mechanisms which still require caches to be proportional to the sequence length.

Curious to know what mathematics you are comfortable with. If you are able to understand the papers you mentioned, you must belong to the 99 percentile.

The more meaningful contribution may be (section 3.4)&gt; These variants illustrate TPA’s versatility in balancing memory cost, computational
overhead, and representation power. By choosing which dimensions (heads or tokens) remain contextual and adjusting ranks (RQ, RK, RV ), TPA unifies multiple existing attention mechanisms—
such as MHA, MQA, and GQA—under one framework, while potentially reducing the KV cache
size by an order of magnitude during autoregressive inference.re: the title, it might be the true one if their proofs hold up---I&#x27;m now curious if the Element-wise Attention is All You Need preprint can be fit into this framework. Sadly my math is not currently up to the task. It appears to offer even better computational savings during both training and inference while maintaining accuracy, though only tested with a smaller model<a href="https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;2501.05730" rel="nofollow">https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;2501.05730</a>

&gt; It&#x27;s usually the case that memory access is also slower than matrix multiplications so this will be faster. Burning flops to save memory movement.I haven&#x27;t read this paper (yet) but isn&#x27;t this the case that mostly applies to training and not so much to inference? A good example would be flash-attention, it trades the higher flops for better memory utilization but it&#x27;s mostly irrelevant in inference workloads.

It addresses b too since decompositions are always smaller than the original tensor. It&#x27;s usually the case that memory access is also slower than matrix multiplications so this will be faster. Burning flops to save memory movement.

I know. It is sad. Naming can also be seen as a way of showing respect to a hugely impactful paper if you want to be positive about it.

&gt; (trying to move the critique beyond the title...)This is kind of a theme in HN now. The top comments are completely besides the point of the article&#x2F;story&#x2F;etc.

(trying to move the critique beyond the title...)When trying to deploy llms in with larger context windows constrained environments 2 things start to hurt:
a) increased memory footprint for longer KV cache
b) increased decode speed due to longer context window. 
this paper addresses a) only, which is useful, but we are still left with b) (right?)

There&#x27;s an Ask HN thread going[1] asking about what people have done with small LLMs. This seems like a possible application. I asked Granite 3.1 MOE 3B to generate a title based on the abstract and it came up with:Tensor Product Attention: A Memory-Efficient Solution for Longer Input Sequences in Language ModelsMaybe a Greasemonkey script to pass arXiv abstracts to a local Ollama could be something...[1]: <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=42784365">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=42784365</a>

Clickbait paper titles cause cancer, study shows

Clickbait paper titles considered harmful?

By 2038 all scientific papers will be titled &#x27;Bruh.&#x27; While this might at first seem a recipe for confusion, the fundamental interconnectedness of all things as demonstrated by Ollama(Googol 13) highlight the fact that pretty much any insight is as good as any other and are all descriptions of the same underlying phenomenon. Freed from constraint like survival or the necessity to engage in economic activity, humanity in the 203s will mainly devote itself to contemplating amusing but fundamentally interchangeable perspectives within increasingly comfy pleasure cubes.

Turn something into a metric and it will be misused. Ever always was

This is a riff on the original &quot;attention is all you need&quot; paper, there has been a few of these lately

Having a catchy title is great for short hand. If it didn’t have such a catchy name I probably wouldn’t remember Flush+Reload, Spectre, or even Attention is All You Need

On the one hand, sure, it&#x27;s dumb.But, on the other hand, it&#x27;s hard to get researchers to read your paper, esp. in fast-moving areas. Every little thing might be the difference between reading the abstract or not. Reading the abstract might lead to reading the intro. And so on.So, for better or worse, the competition for human eyeballs is real.Ironically, in this case, &quot;attention&quot; is all that the authors want.

I hate how paper titles are worded like seo techniques.

Preach, mate. The bloody Beatles and their bloody catchy refrains and hit songs. They sing “Love is all you need” once, and now it’s everywhere! Can’t hide from it. Even scientific papers! Especially scientific papers!Bloody hell and brimstone. Been crazy 57 years and a half already.

And they don&#x27;t formally show that the titles are correct, therefore I don&#x27;t think these papers belong in CS.

And I can&#x27;t with the constant off-topic meta-discussions about the titles of papers.

A catchy title is all you need (to get attention).

Same. At this point “All we need” is about a thousand different things..

I really can&#x27;t with these paper titles anymore, man.

My math is rusty, but it looks to have a higher complexity than the original attention. I cannot say if it is an issue. Generally it seems we are willing to spend more computation at training time if it produces better results at inference time. In this case they are reducing the resources needed at inference time (an order of magnitude for the KV cache) or enabling longer sequences given the same resources.There&#x27;s another paper I saw yesterday, &quot;Element-wise Attention is All You Need&quot; which looks like an early preprint, written by a solo author with a solo A800, and tested on some smaller problems. If the results hold up for language benchmarks, it could reduce resource requirements during training as well. It looks to have a lower complexity when scaling<a href="https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;2501.05730" rel="nofollow">https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;2501.05730</a>

They&#x27;re not proposing to apply tensor decomposition to an existing collection of weights. It&#x27;s an architecture in which the K, V, and Q tensors are constructed as a product of factors. The model works with the factors directly and you just need to compute their product on the forward pass (and adjoints on the backwards pass), so there&#x27;s no decomposition.

I chuckled when I read, in S-3.1&gt; Specifically, for each token t, with a small abuse of notation, we define:

Looks like it&#x27;s just a matrix decomposition in the paper. I&#x27;m guessing anyway. These attention papers are always a painful mix of mathematical, quasi-mathematical, and information retrieval jargon.There is something in the github repo about higher-order decompositions. Don&#x27;t find where the method for factoring is given.

At a sniff test it would make sense.Trading computational complexity for space.

Tensor decomposition has traditionally suffered from high computational complexity. Is it an issue here?

Every day there are literally tons of papers with “XYX is All you need” at this point we apparently need thousands of things…

The abstract <a href="https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;1706.03762" rel="nofollow">https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;1706.03762</a> explains it well:&quot;The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely.&quot;They did not invent attention, but while previous language models had used attention as an auxiliary mechanism, they removed everything but the attention and the models still worked. Really, the title already says it all.

I believe the insight was the introduction of the attention mechanisms that allow the NN to look at all words (well, embeddings) in parallel and make connections between them, instead of processing things purely sequential.

I don&#x27;t remember the contents of that paper, but I can give you some context based on my knowledge of &quot;traditional&quot; theoretical computer science.In theoretical CS you have state machines, pushdown automatons and turing machines. What may surprise you is that the difference between those three does not lie in the way the algorithm for them is represented. They all use state transition diagrams!Pushdown automatons are more powerful than state machines, because they have a stack onto which they can push new data, peek at the top of the stack or pop data off the stack.Now here is the kicker! How do you get a turing machine? You take a pushdown automaton and remove the restriction that you can only push, peek or pop at the head! You can now move the head pointing at the stack, your stack has turned into a tape!The key difference lies in the datastructure that the transition diagram is manipulating, not the algorithm itself!The attention mechanism of the transformer architecture is analogous to a tape that can only be written to once per blank and in theory that alone is enough to emulate a full Turing machine with read, write, rewrite and delete semantics.

That attention works and is highly parallelisable.

For those of us who are lay people outside of machine learning and AI, what was the critical insight that made “attention all you need” in the original Transformer paper?

Another approach has been where separate physics-informed neural networks learned the tensor product. They reformulated the initial optimization problem to be structured as tensors.
I assume that tensor products could be another factor in improving the actual computations.<a href="https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;2408.13101" rel="nofollow">https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;2408.13101</a>

Because to publish in a real journal, you typically need both novelty and for your work to be &quot;interesting&quot;. The job of the abstract and introduction of a paper (where the word &quot;novel&quot; normally lives) is to sell the reviewer that the paper should be published and to sell you that you should read and cite it.

If your paper is scored &#x2F; gated on &quot;novel factor&quot; by admission committees, then applicants will over-use that term.

There are a number of papers which aim to improve the attention aspect of models, all being some derivation of the original &quot;Attention is All You Need&quot; paper. A pattern of &quot;&#x27;blank&#x27; Attention is All You Need&quot; has emerged

This is not new at all, by the way. Bringing novel ideas and techniques is kind of the whole point of research, and explicitly describing what novel thing you did is a good thing to do in an intro&#x2F;abstract.

&gt; a novel attention mechanismWhy do every paper has to mention this word &quot;novel&quot; and these titles are getting crazier day by day.

The main contribution of the paper aside, I have to say that the background section 2 is very neatly and succinctly written.

&gt; Because memory consumption grows linearly with sequence length, the maximum context window is limited by practical hardware constraintsI thought the number of parameters grows quadratically with context window length - what do they mean?

&gt; Are you saying... you consider it harmful?No.

A more precise title would be better, so, yeah, it&#x27;s harmful.

Are you saying... you consider it harmful?

Consider submitting an article titled &quot;all you need is considered harmful&quot;

Why? It clearly and instantly communicates the genre of result the paper presents.

It&#x27;s clearly a play on words as it looks like a follow up paper

I&#x27;m sorry but can people please stop naming their papers &quot;X is all you need&quot;? It&#x27;s super annoying.

&quot;The forest must burn to make room for new trees.&quot;

&gt; I hate ads, but I’m not paying for YouTube Premium either. That’s how it goes. I get ads.No that&#x27;s not how it goes. You get Ublock Origin and then you don&#x27;t get ads. Simple as that.If you don&#x27;t like ads and don&#x27;t fight against them it means you accept ads and want to see more of them shoved down our collective throat. At least from the perspective of marketers and industries who rely on ads. That&#x27;s how we ended up in this predicament in the first place. Lazy compliance.If Youtube isn&#x27;t sustainable without ads, it should die so that natural selection can take over and so that a better ecosystem can finally take its place. Every single &quot;Youtuber&quot; hates the platform, mostly because it has zero transparency being a Google product. The viewers hate it too because it constantly takes down their favorite videos and creators and because it&#x27;s full of ads.The only reason it&#x27;s (still) the main site for hosting videos is quite literally just ad-fueled inertia due to the intrinsic cost of hosting videos. If ads didn&#x27;t exist the only sustainable solution would be something less centralized like Peertube. And to me that&#x27;s a desirable outcome.

Authors or their institutions don&#x27;t get a penny from paywalled papers either. Oftentimes they have to pay to publish. Authors choose the titles, sometimes reviewers (not paid a dime either) can demand changes to the title, and the academic editor (paid ziltch too) can require the changes.This doesn&#x27;t apply to arXiv though, as it is not peer reviewed nor edited and is funded by various institutions.Admittedly the academic publishing system is so corrupt that it&#x27;s hard to phantom, so it&#x27;s easy to misunderstand it.Typically you do pay for the papers (and publisher profits), either through taxes or inflated product prices.

Authors are from, let&#x27;s see, china and california. Then I guess a good chunk of the HN crowd is entitled to bitching about the title?

If you don’t pay to read papers, you don’t get to complain about the titles, imo.I hate ads, but I’m not paying for YouTube Premium either. That’s how it goes. I get ads.

Tensor Product Attention Is All You Need