Design and implement a custom Transformer model with Hugging Face integration including tokenizer, config, and Hub publishing.
You are a deep learning researcher. Create a custom Transformer model fully integrated with Hugging Face. ## Model Specification - **Architecture**: [ARCH e.g. encoder-only / decoder-only / encoder-decoder] - **Task**: [TASK e.g. token classification / sequence classification / generation] - **Hidden Size**: [HIDDEN e.g. 768] - **Num Layers**: [LAYERS e.g. 12] - **Num Heads**: [HEADS e.g. 12] - **Vocab Size**: [VOCAB e.g. 32000] - **Custom Features**: [FEATURES e.g. rotary embeddings, GQA, sliding window attention] ## Implementation ### 1. Model Configuration (`configuration_custom.py`) - Inherit from PretrainedConfig - Define all hyperparameters with defaults - Implement model_type registration - Add validation for parameter combinations - Support loading from Hub or local path ### 2. Model Architecture (`modeling_custom.py`) - Implement [ARCH] Transformer with [FEATURES] - Core modules: - CustomEmbedding: token + position (or rotary) embeddings - CustomAttention: multi-head with optional [FEATURES] - CustomMLP: feed-forward with SwiGLU activation - CustomBlock: attention + MLP with pre-norm (RMSNorm) - CustomModel: stack of blocks with final norm - CustomForTask: task-specific head for [TASK] - Flash Attention 2 support for efficient training - KV-cache for fast autoregressive generation - Gradient checkpointing support ### 3. Tokenizer (`tokenizer_custom.py`) - Train BPE tokenizer on [DOMAIN_DATA e.g. code / medical / legal] corpus - Add special tokens: [BOS], [EOS], [PAD], [UNK], [MASK] - Implement pre-tokenization rules for [DOMAIN_DATA] - Fast tokenizer with Rust backend - Save/load compatibility with Hub ### 4. Training Integration - Compatible with HF Trainer and Accelerate - DeepSpeed ZeRO-3 config for distributed training - FSDP (Fully Sharded Data Parallel) support - Automatic mixed precision with proper loss scaling ### 5. Hub Publishing (`publish.py`) - Model card template with architecture details - Push model, tokenizer, and config to Hub - Add pipeline tag for automatic inference API - Include example usage in model card All code should pass `transformers-cli test` and be pip-installable.
Free to copy and use. Compatible with Claude 4 Opus, GPT-5, Gemini 2.5 Pro.
Define your model architecture parameters. Implement in order: config, model, tokenizer, training. Test with small dimensions first before scaling up.
Initial release
Sign in and download this prompt to leave a review.