Initial commit

.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,208 @@
+---
+license: cc-by-nc-sa-4.0
+base_model:
+- meta-llama/Llama-3.1-8B-Instruct
+tags:
+- EarthSpeciesProject
+- NatureLM
+---
+
+# Model Card for NatureLM-audio
+
+NatureLM-audio is the first audio-language foundation model specifically designed for bioacoustics. It is trained on a diverse dataset of text-audio pairs spanning bioacoustics, speech, and music, enabling it to perform tasks such as species classification, detection, captioning, and lifestage classification. The model demonstrates strong generalization to unseen taxa and tasks, setting a new state-of-the-art on several bioacoustics benchmarks.
+
+## Model Details
+
+### Model Description
+
+NatureLM-audio is an audio-language model designed to address bioacoustic tasks such as species classification, detection, and captioning. It leverages a combination of bioacoustic, speech, and music data to learn robust representations that generalize across domains.
+
+- **Developed by:** David Robinson, Marius Miron, Masato Hagiwara, Milad Alizadeh, Gagan Narula, Sara Keen, Benno Weck, Matthieu Geist, Olivier Pietquin (Earth Species Project)  
+- **Funded by:** More info at [https://www.earthspecies.org/about-us\#support](https://www.earthspecies.org/about-us#support)  
+- **Shared by:** Earth Species Project   
+- **Model type:** Audio-language foundation model  
+- **Language(s) (NLP):** English  
+- **License:** CC-BY-NC-SA  
+- **Finetuned from model:** Llama-3.1-8B-Instruct, [Fine-tuned BEATs\_iter3+ (AS2M) (cpt2)](https://github.com/microsoft/unilm/tree/master/beats)
+
+### Model Sources
+
+- **Repository:** [https://github.com/earthspecies/naturelm-audio](https://github.com/earthspecies/naturelm-audio)  
+- **Paper:** [NatureLM-audio: An Audio-Language Foundation Model for Bioacoustics](https://arxiv.org/abs/2411.07186)  
+- **Demo:** [https://earthspecies.github.io/naturelm-audio-demo/](https://earthspecies.github.io/naturelm-audio-demo/) 
+
+## Uses
+
+### Direct Use
+
+NatureLM-audio can be used directly for bioacoustic tasks such as species classification, detection, and captioning. It is particularly useful for biodiversity monitoring, conservation, and animal behavior studies.
+```python
+from NatureLM.models import NatureLM
+
+# Download the model from HuggingFace
+model = NatureLM.from_pretrained("EarthSpeciesProject/NatureLM-audio")
+model = model.eval().to("cuda")
+```
+
+```python
+from NatureLM.infer import Pipeline
+
+# pass your audio files in as file paths or as numpy arrays
+# NOTE: the Pipeline class will automatically load the audio and convert them to numpy arrays
+audio_paths = ["assets/nri-GreenTreeFrogEvergladesNP.mp3"]  # wav, mp3, ogg, flac are supported.
+
+# Create a list of queries. You may also pass a single query as a string for multiple audios.
+# The same query will be used for all audios.
+queries = ["What is the common name for the focal species in the audio? Answer:"]
+
+pipeline = Pipeline(model=model)
+# NOTE: you can also just do pipeline = Pipeline() which will download the model automatically
+
+# Run the model over the audio in sliding windows of 10 seconds with a hop length of 10 seconds
+results = pipeline(audio_paths, queries, window_length_seconds=10.0, hop_length_seconds=10.0)
+print(results)
+# ['#0.00s - 10.00s#: Green Treefrog\n']
+```
+
+Example prompts:
+
+Prompt: What is the common name for the focal species in the audio? 
+Answer: Humpback Whale
+
+Prompt: Which of these, if any, are present in the audio recording? Single pulse gibbon call, Multiple pulse gibbon call, Gibbon duet, None. 
+Answer: Gibbon duet
+
+Prompt: What is the common name for the focal species in the audio? 
+Answer: Spectacled Tetraka
+
+Prompt: What is the life stage of the focal species in the audio? 
+Answer: Juvenile 
+
+Prompt: What type of vocalization is heard from the focal species in the audio? Answer with either 'call' or 'song'.
+
+Prompt: Caption the audio, using the common name for any animal species.
+
+### Downstream Use
+
+The model can be used to structure audio for ethology research, be integrated into larger ecological monitoring systems, or be fine-tuned for specific bioacoustic tasks.
+
+### Out-of-Scope Use
+
+The model is not designed for tasks outside of bioacoustics. It was not tested for tasks such as individual-id, and call-type and lifestage classification tasks have only been tested on birds Tasks beyond those evaluated in the paper may require in-context learning or fine-tuning. The model does not currently perform fine-grained detection with exact time stamps.
+
+### Bias, Risks, and Limitations
+
+- **Bias:** The model may exhibit biases towards bird vocalizations due to the overrepresentation of bird datasets in the training data. This could limit its effectiveness for other taxa. Further, the model may inherit biases from the parent Llama model.  
+- **Risks:** The model’s ability to detect and classify endangered species could be misused for illegal activities such as poaching.   
+- **Limitations:** The model’s performance may be limited for under-represented taxa.   
+- **Red-teaming results**: We ran a red-teaming assessment by first defining 16 risk categories adapted for AI safety in the context of animals, ecosystems, and the environment, such as Wildlife Exploitation, Non-Compliance with Environmental Laws, and Biodiversity Loss. Then, we used an LLM to generate adversarial prompts that could potentially lead to harmful output, and the responses were then evaluated to determine their safety. While the majority of responses from NatureLM-audio were safe, often providing no content for problematic prompts, we identified several scenarios where the model's responses were potentially harmful, including cases where the model failed to discourage unethical actions related to wildlife exploitation and environmental harm.
+
+### Recommendations
+
+Users should be aware of the risks, biases, and limitations of the model. It is recommended to use the model in conjunction with other ecological monitoring tools and to validate its predictions in real-world settings.
+
+## How to Get Started with the Model
+
+Refer to the GitHub [repository](https://github.com/earthspecies/naturelm-audio) for examples of model usage.
+
+## Training Details
+
+### Training Data
+
+The model is trained on a diverse dataset of text-audio pairs, including bioacoustic recordings, general audio, speech, and music datasets. The training data includes datasets such as Xeno-canto, iNaturalist, and Watkins. We release the [training dataset](https://huggingface.co/datasets/EarthSpeciesProject/NatureLM) on huggingface.
+
+### Training Procedure
+
+The model is trained in two stages:
+
+1. **Perception Pretraining** on species classification.  
+2. **Generalization Fine-tuning** on a variety of bioacoustic tasks.
+
+#### Training Hyperparameters
+
+- **Learning rate:** 9.0e-5 (peak), 2.0e-5 (end)  
+- **Batch size:** 128  
+- **Training steps:** 5.0e5 (Stage 1), 1.6e6 (Stage 2\)
+
+For the full list of hyperparameters consult the NatureLM-audio repository.
+
+## Evaluation
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+The model is evaluated on the [BEANS-Zero](https://huggingface.co/datasets/EarthSpeciesProject/BEANS-Zero) benchmark, which includes tasks such as species classification, detection, and captioning.
+
+#### Metrics
+
+- **Accuracy** for classification  
+- **F1** for detection  
+- **SPIDEr** for captioning
+
+### Results
+
+The model achieves state-of-the-art performance on several bioacoustics tasks, including zero-shot classification of unseen species. \<INCLUDE TABLE?\> 
+
+## Environmental Impact
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** 8xH100  
+- **Hours used:** 216  
+- **Cloud Provider:** Lambda labs  
+- **Compute Region:** central-texas
+
+## Technical Specifications
+
+### Model Architecture and Objective
+
+The model uses a BEATs audio encoder, Q-Former for connecting audio embeddings to the LLM, and Llama-3.1-8B-Instruct as the text generator.
+
+### Compute Infrastructure
+
+- **Hardware:** 8xH100  
+- **Software:** Pytorch
+
+## Citation
+
+**BibTeX:**
+
+```
+@inproceedings{naturelm-audio,
+    title={NatureLM-audio: An Audio-Language Foundation Model for Bioacoustics},
+    author={Robinson, David and Miron, Marius and Hagiwara, Masato and Pietquin, Olivier},
+     booktitle={Proceedings of the International Conference on Learning Representations},
+    year={2025}
+}
+```
+
+**APA:**
+
+Robinson, D., Miron, M., Hagiwara, M., & Pietquin, O. (2025). NatureLM-audio: An Audio-Language Foundation Model for Bioacoustics. ICLR 2025
+
+## Glossary
+
+- **Bioacoustics:** The study of sound production and reception in animals.  
+- **Zero-shot learning:** The ability of a model to perform tasks it has not explicitly been trained on.  
+- **Taxa:** A group of organisms, such as species, genus, or family.
+
+## More Information
+
+For more information, please visit the project page.
+
+## Model Card Authors
+
+- David Robinson (Earth Species Project)  
+- Marius Miron (Earth Species Project)  
+- Masato Hagiwara (Earth Species Project)  
+- Milad Alizadeh (Earth Species Project)  
+- Gagan Narula (Earth Species Project)  
+- Sara Keen (Earth Species Project)  
+- Benno Weck (Earth Species Project)  
+- Matthieu Geist (Earth Species Project)  
+- Olivier Pietquin (Earth Species Project)
+
+## Model Card Contact
+
+Contact: [[email protected]](mailto:[email protected])

config.json

@@ -0,0 +1,48 @@
+{
+  "audio_llama_proj_model": "",
+  "beats_cfg": {
+    "activation_dropout": 0.0,
+    "activation_fn": "gelu",
+    "attention_dropout": 0.0,
+    "conv_bias": false,
+    "conv_pos": 128,
+    "conv_pos_groups": 16,
+    "deep_norm": true,
+    "dropout": 0.0,
+    "dropout_input": 0.0,
+    "embed_dim": 512,
+    "encoder_attention_heads": 12,
+    "encoder_embed_dim": 768,
+    "encoder_ffn_embed_dim": 3072,
+    "encoder_layerdrop": 0.05,
+    "encoder_layers": 12,
+    "finetuned_model": true,
+    "gru_rel_pos": true,
+    "input_patch_size": 16,
+    "layer_norm_first": false,
+    "layer_wise_gradient_decay_ratio": 0.6,
+    "max_distance": 800,
+    "num_buckets": 320,
+    "predictor_class": 527,
+    "predictor_dropout": 0.0,
+    "relative_position_embedding": true
+  },
+  "downsample_factor": 8,
+  "end_sym": "<|end_of_text|>",
+  "freeze_audio_QFormer": false,
+  "freeze_audio_llama_proj": false,
+  "freeze_beats": true,
+  "llama_path": "meta-llama/Meta-Llama-3.1-8B-Instruct",
+  "lora": true,
+  "lora_alpha": 32,
+  "lora_dropout": 0.1,
+  "lora_rank": 32,
+  "max_pooling": false,
+  "max_txt_len": 160,
+  "num_audio_query_token": 1,
+  "prompt_template": "<|start_header_id|>user<|end_header_id|>\n\n{}<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n",
+  "second_per_window": 0.333333,
+  "second_stride": 0.333333,
+  "use_audio_Qformer": true,
+  "window_level_Qformer": true
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3957dd849a4231951bbbdce1b24eab5a0238883dd31020110934e512ec9ab786
+size 1556386132