README.md

metadata

title: DLRNA-BERTa9700 Interface
sdk: gradio
sdk_version: 5.44.0
app_file: app.py
pinned: false

Drug-target interaction prediction model

Model description

This model predicts drug-target interactions using a novel cross-attention architecture that combines RNA sequence understanding with molecular representation learning. The model processes RNA target sequences and drug SMILES representations to predict binding affinity scores (pKd values).

Architecture

The model consists of several key components:

1. Target encoder: RNA-BERTa model that processes RNA sequences (nucleotides A, U, G, C)
2. Drug encoder: ChemBERTa-77M-MTR model [1] that processes molecular SMILES representations
3. Cross-attention layer: Single-head attention mechanism (1 head) that models interactions between drug and target representations
4. Regression head: Predicts binding affinity scores with learnable scaling and bias parameters

Technical specifications

• Model size: Combines RNA-BERTa (target encoder) + ChemBERTa-77M-MTR (drug encoder)
• Cross-attention: Single-head attention with 384-dimensional embeddings
• Maximum sequence length: 512 tokens for both target and drug inputs
• Output: Continuous binding affinity prediction (pKd values)
• Dropout: Configurable attention dropout and hidden dropout for regularization
• Layer normalization: Applied for training stability

Performance metrics

Evaluated on external ROBIN test datasets [2] across different RNA classes:

Dataset	Precision	Specificity	Recall	AUROC	F1 Score
Aptamers	0.648	0.002	1.000	0.571	0.787
Riboswitch	0.519	0.035	0.972	0.577	0.677
Viral RNA	0.562	0.095	0.943	0.579	0.704
miRNA	0.373	0.028	0.991	0.596	0.542

Usage

Using the Gradio interface

import gradio as gr
from updated_app import demo

# Launch the interactive interface
demo.launch()

Programmatic usage

from modeling_dlmberta import InteractionModelATTNForRegression, StdScaler
from configuration_dlmberta import InteractionModelATTNConfig
from transformers import AutoModel, RobertaModel, AutoConfig
from chemberta import ChembertaTokenizer

# Load model components
config = InteractionModelATTNConfig.from_pretrained("path/to/model")

# Load encoders
target_encoder = AutoModel.from_pretrained("IlPakoZ/RNA-BERTa9700")
drug_encoder_config = AutoConfig.from_pretrained("DeepChem/ChemBERTa-77M-MTR")
drug_encoder_config.pooler = None
drug_encoder = RobertaModel(config=drug_encoder_config, add_pooling_layer=False)

# Load scaler (if available)
scaler = StdScaler()
scaler.load("path/to/model")

# Initialize model
model = InteractionModelATTNForRegression.from_pretrained(
    "path/to/model",
    config=config,
    target_encoder=target_encoder,
    drug_encoder=drug_encoder,
    scaler=scaler
)

# Make predictions
target_sequence = "AUGCGAUCGACGUACGUUAGCCGUAGCGUAGCUAGUGUAGCUAGUAGCU"
drug_smiles = "C1=CC=C(C=C1)NC(=O)C2=CC=CC=N2"

# Tokenize inputs
target_inputs = target_tokenizer(target_sequence, padding="max_length", truncation=True, max_length=512, return_tensors="pt")
drug_inputs = drug_tokenizer(drug_smiles, padding="max_length", truncation=True, max_length=512, return_tensors="pt")

# Predict
with torch.no_grad():
    prediction = model(target_inputs, drug_inputs)
    if model.scaler:
        prediction = model.unscale(prediction)

Model inputs

• Target sequence: RNA sequence using nucleotides A, U, G, C (string)
• Drug SMILES: Simplified Molecular Input Line Entry System notation (string)

Model outputs

• Binding affinity: Predicted pKd binding affinity score (float)
• Attention weights: Cross-attention weights for interpretability analysis (when enabled)

Interpretability features

The model includes advanced interpretability capabilities:

• Cross-attention visualization: Heatmaps showing interaction patterns between drug and target tokens
• Token-level contributions: Visualization of individual token contributions to the final prediction
• Unnormalized vs. normalized contributions: Both scaled and unscaled contribution analysis
• Interpretation mode: Special mode for extracting attention weights and intermediate values

Enabling interpretation mode

# Enable interpretation mode (evaluation only)
model.INTERPR_ENABLE_MODE()

# Make prediction with interpretation data
prediction = model(target_inputs, drug_inputs)

# Access interpretation data
cross_attention_weights = model.model.crossattention_weights
presum_contributions = model.model.presum_layer
attention_scores = model.model.scores

# Disable interpretation mode
model.INTERPR_DISABLE_MODE()

Training details

Data processing

• Scaling: Uses StdScaler for target value normalization
• Tokenization: Separate tokenizers for RNA sequences and SMILES strings
• Padding: Max length padding to 512 tokens
• Masking: Attention masks to handle variable-length sequences

Architecture details

• Embedding dimension: 384 for cross-attention layer
• Target encoder output: 512 dimensions, mapped to 384
• Drug encoder output: 384 dimensions (direct use)
• Attention mechanism: Single-head cross-attention with scaled dot-product
• Learnable parameters: Weighted sum with learnable scaling vector and bias
• Padding handling: Learnable padding value for masked positions

Limitations

• Performance varies significantly across RNA classes (miRNA shows lower precision)
• May not generalize well to RNA sequences or chemical scaffolds not represented in training data
• Computational requirements scale with sequence length (max 512 tokens)
• Single attention head may limit capacity to capture diverse interaction patterns
• SMILES representation may not capture all relevant molecular properties

Files in this repository

• modeling_dlmberta.py: Main model implementation with cross-attention architecture
• configuration_dlmberta.py: Model configuration class
• chemberta.py: Custom tokenizer for chemical SMILES processing
• updated_app.py: Gradio application interface with visualization capabilities
• analysis.py: Visualization functions for interpretability
• requirements.txt: Python dependencies
• config.json: Model configuration file

License

This model is released under the MIT License.

Citations

[1]

@article{ahmad2022chemberta,
  title={Chemberta-2: Towards chemical foundation models},
  author={Ahmad, Walid and Simon, Elana and Chithrananda, Seyone and Grand, Gabriel and Ramsundar, Bharath},
  journal={arXiv preprint arXiv:2209.01712},
  year={2022}
}

[2]

@article{krishnan2024reliable,
  title={Reliable method for predicting the binding affinity of RNA-small molecule interactions using machine learning},
  author={Krishnan, Sowmya R and Roy, Arijit and Gromiha, M Michael},
  journal={Briefings in Bioinformatics},
  volume={25},
  number={2},
  pages={bbae002},
  year={2024},
  publisher={Oxford University Press}
}