Upload Multi-Token Prediction GPT-2 model

README.md

@@ -0,0 +1,96 @@
+---
+language: en
+license: apache-2.0
+library_name: transformers
+tags:
+- multi-token-prediction
+- gpt2
+- mathematics
+- MetaMathQA
+- transformer
+- speculative-decoding
+pipeline_tag: text-generation
+base_model: gpt2
+---
+
+# Multi-Token Prediction GPT-2 Model
+
+This is a GPT-2 model enhanced with **Multi-Token Prediction (MTP)** architecture, trained on the MetaMathQA dataset for mathematical reasoning tasks.
+
+## Model Description
+
+This model implements the Multi-Token Prediction approach from the paper "Better & Faster Large Language Models via Multi-token Prediction" by Meta AI. Key features:
+
+- **Shared Trunk Architecture**: Uses 9 shared transformer layers with 4 prediction heads
+- **Multi-Token Prediction**: Predicts 4 tokens simultaneously (t+1, t+2, t+3, t+4)
+- **Enhanced Speculative Decoding**: Achieves up to 3x inference speedup
+- **Mathematical Reasoning**: Fine-tuned specifically for mathematical problem solving
+
+## Architecture Details
+
+- **Base Model**: GPT-2 (124M parameters)
+- **Trunk Layers**: 9 (shared processing)
+- **Prediction Heads**: 4 parallel heads
+- **Training Data**: MetaMathQA dataset (500 samples)
+- **Training Epochs**: 1 with gradient accumulation
+
+## Usage
+
+```python
+import torch
+from transformers import GPT2Tokenizer
+# Note: You'll need the custom MultiTokenGPT2 class from the training code
+
+tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
+tokenizer.pad_token = tokenizer.eos_token
+
+# Load your model (custom loading required)
+# model = MultiTokenGPT2.from_pretrained("Goldenwert/multitoken-gpt2-metamathqa")
+
+prompt = "What is the derivative of f(x) = x^3?"
+input_ids = tokenizer.encode(prompt, return_tensors="pt")
+
+# Standard generation
+generated = model.generate(
+    input_ids,
+    max_new_tokens=50,
+    use_speculative=False
+)
+
+# Fast speculative generation
+generated_fast = model.generate(
+    input_ids,
+    max_new_tokens=50,
+    use_speculative=True
+)
+
+print(tokenizer.decode(generated[0], skip_special_tokens=True))
+```
+
+## Performance
+
+- **Inference Speed**: Up to 3x faster with speculative decoding
+- **Memory Efficiency**: Gradient checkpointing support
+- **Mathematical Tasks**: Improved reasoning on math problems
+
+## Training Details
+
+- **Dataset**: MetaMathQA (mathematical reasoning)
+- **Optimizer**: AdamW with warmup
+- **Learning Rate**: 5e-5
+- **Batch Size**: 4 (effective 32 with gradient accumulation)
+- **Hardware**: GPU with FP16 precision
+
+## Research Context
+
+Based on the paper "Better & Faster Large Language Models via Multi-token Prediction" which demonstrates that predicting multiple tokens simultaneously can improve both training efficiency and inference speed.
+
+## Files
+
+- `pytorch_model.bin`: Model weights
+- `config.json`: Model configuration
+- Additional training artifacts
+
+## License
+
+Apache 2.0 (following GPT-2 base model)

checkpoint-11110/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c80bae54059dc062c6252892a9753c0833081fa2bba482a80753ae2517c3134b
+size 1052355322

checkpoint-11110/pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bd0a751af18be5499e755deec78668429a11049674ad4462213470816df7bbf9
+size 526171530

checkpoint-11110/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:40fc25e8ba1bd308447587d2888737af7602efd0bad1a094cfc837d565cbdafa
+size 14244

checkpoint-11110/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c13eaf49e2347ca3e373d8fa1ac3b0078f46477102d5bac9947d4e1d1f45a43e
+size 1064

checkpoint-11110/special_tokens_map.json

@@ -0,0 +1,6 @@
+{
+  "bos_token": "<|endoftext|>",
+  "eos_token": "<|endoftext|>",
+  "pad_token": "<|endoftext|>",
+  "unk_token": "<|endoftext|>"
+}

checkpoint-11110/tokenizer_config.json

@@ -0,0 +1,21 @@
+{
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "50256": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<|endoftext|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "extra_special_tokens": {},
+  "model_max_length": 1024,
+  "pad_token": "<|endoftext|>",
+  "tokenizer_class": "GPT2Tokenizer",
+  "unk_token": "<|endoftext|>"
+}

checkpoint-11110/trainer_state.json

@@ -0,0 +1,188 @@
+{
+  "best_global_step": null,
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 1.0,
+  "eval_steps": 500,
+  "global_step": 11110,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.0450070323488045,
+      "grad_norm": 1.5042104721069336,
+      "learning_rate": 4.7754275427542756e-05,
+      "loss": 2.0652,
+      "step": 500
+    },
+    {
+      "epoch": 0.090014064697609,
+      "grad_norm": 1.1722426414489746,
+      "learning_rate": 4.55040504050405e-05,
+      "loss": 1.5388,
+      "step": 1000
+    },
+    {
+      "epoch": 0.1350210970464135,
+      "grad_norm": 1.0186536312103271,
+      "learning_rate": 4.325382538253826e-05,
+      "loss": 1.3771,
+      "step": 1500
+    },
+    {
+      "epoch": 0.180028129395218,
+      "grad_norm": 0.9214487075805664,
+      "learning_rate": 4.1003600360036004e-05,
+      "loss": 1.2963,
+      "step": 2000
+    },
+    {
+      "epoch": 0.2250351617440225,
+      "grad_norm": 0.8969003558158875,
+      "learning_rate": 3.875337533753375e-05,
+      "loss": 1.2492,
+      "step": 2500
+    },
+    {
+      "epoch": 0.270042194092827,
+      "grad_norm": 0.8213639259338379,
+      "learning_rate": 3.6503150315031505e-05,
+      "loss": 1.2009,
+      "step": 3000
+    },
+    {
+      "epoch": 0.3150492264416315,
+      "grad_norm": 0.8741646409034729,
+      "learning_rate": 3.425292529252925e-05,
+      "loss": 1.1572,
+      "step": 3500
+    },
+    {
+      "epoch": 0.360056258790436,
+      "grad_norm": 0.7562123537063599,
+      "learning_rate": 3.2002700270027e-05,
+      "loss": 1.1338,
+      "step": 4000
+    },
+    {
+      "epoch": 0.4050632911392405,
+      "grad_norm": 0.8160617351531982,
+      "learning_rate": 2.9752475247524754e-05,
+      "loss": 1.1104,
+      "step": 4500
+    },
+    {
+      "epoch": 0.450070323488045,
+      "grad_norm": 0.790651261806488,
+      "learning_rate": 2.75022502250225e-05,
+      "loss": 1.0986,
+      "step": 5000
+    },
+    {
+      "epoch": 0.49507735583684953,
+      "grad_norm": 0.667475163936615,
+      "learning_rate": 2.525202520252025e-05,
+      "loss": 1.0757,
+      "step": 5500
+    },
+    {
+      "epoch": 0.540084388185654,
+      "grad_norm": 0.7328142523765564,
+      "learning_rate": 2.3001800180018002e-05,
+      "loss": 1.0592,
+      "step": 6000
+    },
+    {
+      "epoch": 0.5850914205344585,
+      "grad_norm": 0.6897611618041992,
+      "learning_rate": 2.0751575157515752e-05,
+      "loss": 1.0475,
+      "step": 6500
+    },
+    {
+      "epoch": 0.630098452883263,
+      "grad_norm": 0.7082722783088684,
+      "learning_rate": 1.85013501350135e-05,
+      "loss": 1.0428,
+      "step": 7000
+    },
+    {
+      "epoch": 0.6751054852320675,
+      "grad_norm": 0.7828446626663208,
+      "learning_rate": 1.625112511251125e-05,
+      "loss": 1.0318,
+      "step": 7500
+    },
+    {
+      "epoch": 0.720112517580872,
+      "grad_norm": 0.8432497978210449,
+      "learning_rate": 1.4000900090009e-05,
+      "loss": 1.024,
+      "step": 8000
+    },
+    {
+      "epoch": 0.7651195499296765,
+      "grad_norm": 0.7583742737770081,
+      "learning_rate": 1.1750675067506751e-05,
+      "loss": 1.0168,
+      "step": 8500
+    },
+    {
+      "epoch": 0.810126582278481,
+      "grad_norm": 0.7724961042404175,
+      "learning_rate": 9.500450045004502e-06,
+      "loss": 1.0015,
+      "step": 9000
+    },
+    {
+      "epoch": 0.8551336146272855,
+      "grad_norm": 0.7692517638206482,
+      "learning_rate": 7.250225022502251e-06,
+      "loss": 1.0006,
+      "step": 9500
+    },
+    {
+      "epoch": 0.90014064697609,
+      "grad_norm": 0.6760990619659424,
+      "learning_rate": 5e-06,
+      "loss": 1.0056,
+      "step": 10000
+    },
+    {
+      "epoch": 0.9451476793248945,
+      "grad_norm": 0.7823421955108643,
+      "learning_rate": 2.7497749774977497e-06,
+      "loss": 1.0042,
+      "step": 10500
+    },
+    {
+      "epoch": 0.9901547116736991,
+      "grad_norm": 0.8093355298042297,
+      "learning_rate": 4.995499549954995e-07,
+      "loss": 0.9944,
+      "step": 11000
+    }
+  ],
+  "logging_steps": 500,
+  "max_steps": 11110,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 1,
+  "save_steps": 99999999,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": true
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 0.0,
+  "train_batch_size": 4,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-11110/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e6a677d8267e403ee6e61f8cd7dcd956d32959303d2ab209618047ecbea6b4a5
+size 5240

checkpoint-15/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d76926653442a6cd13969aa536a82701b4d0c00935170e3ec7223339dc329684
+size 1052355322

checkpoint-15/pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5f88f9624f6492417d1b97ceb06a3d5f2dc764ae6a22865ddd694e73bd538487
+size 526173130

checkpoint-15/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9ecc4530a32c803eee1dbb37f345e40c4c7380974a224f6b8e652a052311d9d2
+size 14244

checkpoint-15/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2d0facd13ec6092c2b43cce23b0a73463bc00999cd16da7cbf795867dc294419
+size 1064

checkpoint-15/special_tokens_map.json

@@ -0,0 +1,6 @@
+{
+  "bos_token": "<|endoftext|>",
+  "eos_token": "<|endoftext|>",
+  "pad_token": "<|endoftext|>",
+  "unk_token": "<|endoftext|>"
+}

checkpoint-15/tokenizer_config.json

@@ -0,0 +1,21 @@
+{
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "50256": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<|endoftext|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "extra_special_tokens": {},
+  "model_max_length": 1024,
+  "pad_token": "<|endoftext|>",
+  "tokenizer_class": "GPT2Tokenizer",
+  "unk_token": "<|endoftext|>"
+}

checkpoint-15/trainer_state.json

@@ -0,0 +1,33 @@
+{
+  "best_global_step": null,
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 1.0,
+  "eval_steps": 500,
+  "global_step": 15,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [],
+  "logging_steps": 500,
+  "max_steps": 15,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 1,
+  "save_steps": 99999999,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": true
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 0.0,
+  "train_batch_size": 4,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-15/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e6a677d8267e403ee6e61f8cd7dcd956d32959303d2ab209618047ecbea6b4a5
+size 5240

config.json

@@ -0,0 +1,37 @@
+{
+  "activation_function": "gelu_new",
+  "architectures": [
+    "GPT2LMHeadModel"
+  ],
+  "attn_pdrop": 0.1,
+  "bos_token_id": 50256,
+  "embd_pdrop": 0.1,
+  "eos_token_id": 50256,
+  "initializer_range": 0.02,
+  "layer_norm_epsilon": 1e-05,
+  "model_type": "gpt2",
+  "n_ctx": 1024,
+  "n_embd": 768,
+  "n_head": 12,
+  "n_inner": null,
+  "n_layer": 9,
+  "n_positions": 1024,
+  "reorder_and_upcast_attn": false,
+  "resid_pdrop": 0.1,
+  "scale_attn_by_inverse_layer_idx": false,
+  "scale_attn_weights": true,
+  "summary_activation": null,
+  "summary_first_dropout": 0.1,
+  "summary_proj_to_labels": true,
+  "summary_type": "cls_index",
+  "summary_use_proj": true,
+  "task_specific_params": {
+    "text-generation": {
+      "do_sample": true,
+      "max_length": 50
+    }
+  },
+  "transformers_version": "4.51.3",
+  "use_cache": false,
+  "vocab_size": 50257
+}

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bd0a751af18be5499e755deec78668429a11049674ad4462213470816df7bbf9
+size 526171530

special_tokens_map.json

@@ -0,0 +1,6 @@
+{
+  "bos_token": "<|endoftext|>",
+  "eos_token": "<|endoftext|>",
+  "pad_token": "<|endoftext|>",
+  "unk_token": "<|endoftext|>"
+}

tokenizer_config.json

@@ -0,0 +1,21 @@
+{
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "50256": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<|endoftext|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "extra_special_tokens": {},
+  "model_max_length": 1024,
+  "pad_token": "<|endoftext|>",
+  "tokenizer_class": "GPT2Tokenizer",
+  "unk_token": "<|endoftext|>"
+}

training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e6a677d8267e403ee6e61f8cd7dcd956d32959303d2ab209618047ecbea6b4a5
+size 5240