Cannot implement LoRA on a custom model containing transformer encoder from pytorch

[wsuSaiman]

I am using LoRA for my custom model. Within the model i have trasnformer encoder block and a series of linear layers. I want to implement LoRA in the custom model specifically targeting the q,k and v projection weights on the self attention block. However, in the transformer encoder layer in pytorch, I cannot find any module names corresponding to the q, k and v projection.

class CustomModel(nn.Module):
   def __init__():
      self.embedding = nn.Linear(100,512)
      self.encoder_layer_1 = nn.TransformerEncoderLayer(d_model=self.model_dim, nhead=16)
      self.transformers_1 = torch.nn.TransformerEncoder(self.encoder_layer_1 , num_layers = 24)
      self.output = nn.Linear(200 * 512,200)

     def forward():

The code above represents the architecture of my custom model.

Has anybody implemented LoRA on the nn.Transformer on the q,k and v projection weights before? If yes, what is the module name for those weights?