MergedLinear still available?

[DanielRoeder1]

Feature request

The original lora library as well as some issues of peft show the existence of a MergedLinear layer for models like GPT2 that used a single Linear layer for qkv projection.
The feature seemed to be available some time ago but I can't seem to figure out how to enable it / if it is still available. Initializing a GPT2 model with a LoraConfig now appears to create a normal Lora Lienar layer that does not separate the projections.

Please advise if this feature is still available.

Motivation

MergedLinear is used for Decoder with a shared qkv projection layer.

Your contribution

(This feature was available some time ago, so I am uncertain about the current state=

[younesbelkada]

Hi @DanielRoeder1
Thanks for the issue, per my understanding, the support of MergedLinear has been dropped with the introduction of multi-adapter feature: #263

[DanielRoeder1]

Thanks for the quick reply!
Is there a way to replicate the usage of a MergedLinear layer using these multi-adapters?

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.