Implementation of "Breaking the Low-Rank Dilemma of Linear Attention" The Softmax attention mechanism in Transformer models is notoriously computationally expensive, particularly due to its quadratic ...

Don't be Dense, SLiCE the Cost! Structured Linear Controlled Differential Equations (SLiCEs) are a new class of sequence models that combine the maximal expressivity (i.e., universality) of dense, ...

Abstract: Code representation learning is an important way to encode the semantics of source code through pre-training. The learned representation supports a variety of downstream tasks, such as ...

Abstract: We present SEMANTIC CODE FINDER, a framework for semantic code search that delivers high-level search performance and supports multiple programming languages. Leveraging code summaries, it ...