Low-Rank Adaptation (LoRA) is a reparametrization method that aims to reduce the number of trainable parameters with low-rank representations. The weight matrix is broken down into low-rank matrices that are trained and updated. All the pretrained model parameters remain frozen. After training, the low-rank matrices are added back to the original weights. This makes it more efficient to store and train a LoRA model because there are significantly fewer parameters.
The tags values are defined in the label ids dictionary. The letter that prefixes each label indicates the token position: B is for the first token of an entity, I is for a token inside the entity, and 0 is for a token that is not part of an entity.
Then load the seqeval framework which includes several metrics - precision, accuracy, F1, and recall - for evaluating sequence labeling tasks.
Copied
seqeval = evaluate.load("seqeval")
Now you can write an evaluation function to compute the metrics from the model predictions and labels, and return the precision, recall, F1, and accuracy scores:
Copied
label_list = [
"O",
"B-DNA",
"I-DNA",
"B-protein",
"I-protein",
"B-cell_type",
"I-cell_type",
"B-cell_line",
"I-cell_line",
"B-RNA",
"I-RNA",
]
def compute_metrics(p):
predictions, labels = p
predictions = np.argmax(predictions, axis=2)
true_predictions = [
[label_list[p] for (p, l) in zip(prediction, label) if l != -100]
for prediction, label in zip(predictions, labels)
]
true_labels = [
[label_list[l] for (p, l) in zip(prediction, label) if l != -100]
for prediction, label in zip(predictions, labels)
]
results = seqeval.compute(predictions=true_predictions, references=true_labels)
return {
"precision": results["overall_precision"],
"recall": results["overall_recall"],
"f1": results["overall_f1"],
"accuracy": results["overall_accuracy"],
}
Preprocess dataset
Initialize a tokenizer and make sure you set is_split_into_words=True because the text sequence has already been split into words. However, this doesn’t mean it is tokenized yet (even though it may look like it!), and you’ll need to further tokenize the words into subwords.
Now you’re ready to create a PeftModel. Start by loading the base roberta-large model, the number of expected labels, and the id2label and label2id dictionaries:
lora_alpha, scaling factor for the weight matrices
lora_dropout, dropout probability of the LoRA layers
bias, set to all to train all bias parameters
💡 The weight matrix is scaled by lora_alpha/r, and a higher lora_alpha value assigns more weight to the LoRA activations. For performance, we recommend setting bias to None first, and then lora_only, before trying all.
Pass the base model and peft_config to the get_peft_model() function to create a PeftModel. You can check out how much more efficient training the PeftModel is compared to fully training the base model by printing out the trainable parameters:
Copied
model = get_peft_model(model, peft_config)
model.print_trainable_parameters()
"trainable params: 1855499 || all params: 355894283 || trainable%: 0.5213624069370061"
From the 🌍 Transformers library, create a TrainingArguments class and specify where you want to save the model to, the training hyperparameters, how to evaluate the model, and when to save the checkpoints:
Pass the model, TrainingArguments, datasets, tokenizer, data collator and evaluation function to the Trainer class. The Trainer handles the training loop for you, and when you’re ready, call train to begin!
Once training is complete, you can store and share your model on the Hub if you’d like. Log in to your BOINC AI account and enter your token when prompted:
Copied
from boincai_hub import notebook_login
notebook_login()
Upload the model to a specific model repository on the Hub with the push_to_hub method:
text = "The activation of IL-2 gene expression and NF-kappa B through CD28 requires reactive oxygen production by 5-lipoxygenase."
inputs = tokenizer(text, return_tensors="pt")
Pass the inputs to the model, and print out the model prediction for each token:
