bisectgroup/BiCA-small

SentenceTransformer based on thenlper/gte-small

This is a sentence-transformers model finetuned from thenlper/gte-small. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: thenlper/gte-small
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 384 dimensions
• Similarity Function: Cosine Similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False, 'architecture': 'BertModel'})
  (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    'Neural correlates of advice evaluation and integration in the judge-advisor paradigm',
    'Considering advice from others is a pervasive element of human social life. We used the judge-advisor paradigm to investigate the neural correlates of advice evaluation and advice integration by means of functional magnetic resonance imaging. Our results demonstrate that evaluating advice recruits the "mentalizing network," brain regions activated when people think about others\' mental states. Important activation differences exist, however, depending upon the perceived competence of the advisor. Consistently, additional analyses demonstrate that integrating others\' advice, i.e., how much participants actually adjust their initial estimate, correlates with neural activity in the centromedial amygdala in the case of a competent and with activity in visual cortex in the case of an incompetent advisor. Taken together, our findings, therefore, demonstrate that advice evaluation and integration rely on dissociable neural mechanisms and that significant differences exist depending upon the advisor\'s reputation, which suggests different modes of processing advice depending upon the perceived competence of the advisor.',
    'Humans regulate intergroup conflict through parochial altruism; they self-sacrifice to contribute to in-group welfare and to aggress against competing out-groups. Parochial altruism has distinct survival functions, and the brain may have evolved to sustain and promote in-group cohesion and effectiveness and to ward off threatening out-groups. Here, we have linked oxytocin, a neuropeptide produced in the hypothalamus, to the regulation of intergroup conflict. In three experiments using double-blind placebo-controlled designs, male participants self-administered oxytocin or placebo and made decisions with financial consequences to themselves, their in-group, and a competing out-group. Results showed that oxytocin drives a "tend and defend" response in that it promoted in-group trust and cooperation, and defensive, but not offensive, aggression toward competing out-groups.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 384]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.9575, 0.8147],
#         [0.9575, 1.0000, 0.8303],
#         [0.8147, 0.8303, 1.0000]])

Training Details

Training Dataset

Unnamed Dataset

• Size: 129,971 training samples
• Columns: sentence_0, sentence_1, and sentence_2
• Approximate statistics based on the first 1000 samples:

  	sentence_0	sentence_1	sentence_2
  type	string	string	string
  details	min: 6 tokens mean: 19.55 tokens max: 56 tokens	min: 3 tokens mean: 210.7 tokens max: 512 tokens	min: 24 tokens mean: 312.31 tokens max: 512 tokens

• Samples:

  sentence_0	sentence_1	sentence_2
  Microbiology and immunomics in male infertility	Up to 50% of infertility is caused by the male side. Varicocele, orchitis, prostatitis, oligospermia, asthenospermia, and azoospermia are common causes of impaired male reproductive function and male infertility. In recent years, more and more studies have shown that microorganisms play an increasingly important role in the occurrence of these diseases. This review will discuss the microbiological changes associated with male infertility from the perspective of etiology, and how microorganisms affect the normal function of the male reproductive system through immune mechanisms. Linking male infertility with microbiome and immunomics can help us recognize the immune response under different disease states, providing more targeted immune target therapy for these diseases, and even the possibility of combined immunotherapy and microbial therapy for male infertility.	There are currently no sensitive and specific assays for activin B that could be utilized to study human biological fluids. The aim of this project was to develop and validate a 'total' activin B ELISA for use with human biological fluids and establish concentrations of activin B in the circulation and fluids from the reproductive organs. The new ELISA was validated and then used to measure activin B levels in the circulation of healthy participants, IVF patients, pregnant women and in ovarian follicular fluid and seminal plasma. Healthy adult subjects (n = 143), subjects from an IVF clinic (n = 27) and pregnancy groups (n = 29) were sampled. The sensitivity of the assay was 0.019 ng/ml. Validation of the activin B ELISA showed good recovery (90.7 +/- 9.8%) and linearity in biological fluid and cell culture media and low cross-reactivity with related analytes (inhibin B = 0.077% and activin A = 0.0034%). There was a negative correlation between activin B concentration (r = -0.281, P < ...
  Biomarkers of heterogeneity in type 1 diabetes	The 'Biomarkers of heterogeneity in type 1 diabetes' study cohort was set up to identify genetic, physiological and psychosocial factors explaining the observed heterogeneity in disease progression and the development of complications in people with long-standing type 1 diabetes (T1D).	In patients with type 1 diabetes, there has been concern about the effects of recurrent hypoglycaemia and chronic hyperglycaemia on cognitive function. Because other biomedical factors may also increase the risk of cognitive decline, this study examined whether macrovascular risk factors (hypertension, smoking, hypercholesterolaemia, obesity), sub-clinical macrovascular disease (carotid intima-media thickening, coronary calcification) and microvascular complications (retinopathy, nephropathy) were associated with decrements in cognitive function over an extended time period. Type 1 diabetes patients (n = 1,144) who had completed a comprehensive cognitive test battery at entry into the Diabetes Control and Complications Trial were re-assessed at a mean of 18.5 (range: 15-23) years later. Univariate and multivariable models examined the relationship between cognitive change and the presence of micro- and macrovascular complications and risk factors. Univariate modelling showed that smoki...
  Role of Molecular Profiling and Subgroups in Pediatric Medulloblastoma	As advances in the molecular and genetic profiling of pediatric medulloblastoma evolve, associations with prognosis and treatment are found (prognostic and predictive biomarkers) and research is directed at molecular therapies. Medulloblastoma typically affects young patients, where the implications of any treatment on the developing brain must be carefully considered. The aim of this article is to provide a clear comprehensible update on the role molecular profiling and subgroups in pediatric medulloblastoma as it is likely to contribute significantly toward prognostication. Knowledge of this classification is of particular interest because there are new molecular therapies targeting the Shh subgroup of medulloblastomas.	The Wnt/beta-catenin pathway plays important roles during embryonic development and growth control. The B56 regulatory subunit of protein phosphatase 2A (PP2A) has been implicated as a regulator of this pathway. However, this has not been investigated by loss-of-function analyses. Here we report loss-of-function analysis of PP2A:B56epsilon during early Xenopus embryogenesis. We provide direct evidence that PP2A:B56epsilon is required for Wnt/beta-catenin signaling upstream of Dishevelled and downstream of the Wnt ligand. We show that maternal PP2A:B56epsilon function is required for dorsal development, and PP2A:B56epsilon function is required later for the expression of the Wnt target gene engrailed, for subsequent midbrain-hindbrain boundary formation, and for closure of the neural tube. These data demonstrate a positive role for PP2A:B56epsilon in the Wnt pathway.

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 32
• per_device_eval_batch_size: 32
• num_train_epochs: 1
• max_steps: 20
• multi_dataset_batch_sampler: round_robin

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: no
• prediction_loss_only: True
• per_device_train_batch_size: 32
• per_device_eval_batch_size: 32
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 5e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1
• num_train_epochs: 1
• max_steps: 20
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.0
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: round_robin
• router_mapping: {}
• learning_rate_mapping: {}

Framework Versions

• Python: 3.10.14
• Sentence Transformers: 5.0.0
• Transformers: 4.53.2
• PyTorch: 2.6.0+cu124
• Accelerate: 1.6.0
• Datasets: 3.6.0
• Tokenizers: 0.21.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}