nyuuzyou's picture

Building on HF

nyuuzyou PRO

nyuuzyou

·

https://ducks.party/donate

AI & ML interests

None yet

Recent Activity

reacted to codelion's post with 🔥 about 4 hours ago

Introducing Dhara-70M: A diffusion language model that achieves 3.8x higher throughput than autoregressive models! Key findings from our research on optimal architectures for small language models: → Depth beats width: 32 layers outperforms 12 layers at the same parameter count → Best-in-class factuality: 47.5% on TruthfulQA → 10x training efficiency using WSD (Warmup-Stable-Decay) conversion → Canon layers add only 0.13% parameters but improve reasoning We trained on 1B tokens using the optimal 50-30-20 dataset mix (PDFs + filtered web + educational content), then converted to diffusion with just 100M additional tokens. Blog: https://huggingface.co/blog/codelion/optimal-model-architecture Model: https://huggingface.co/codelion/dhara-70m

posted an update about 9 hours ago

🇨🇳 Gitee Code Dataset - The Missing Piece of the Stack https://huggingface.co/datasets/nyuuzyou/gitee-code Gitee is not included in the Software Heritage archive, meaning it is currently missing from datasets like The Stack. This release fills that massive gap, serving as the largest Chinese code dataset and one of the largest code corpuses overall. - 819,472,785 files from 3,105,923 repositories - 536 GB compressed Parquet storage - 554 programming languages - Extensive quality filtering: Removed vendor code, artifacts, and generated files - Rich Chinese language understanding: High volume of Chinese comments and docs Huge thanks to Hugging Face for the storage grant that made hosting this (and all my other datasets) possible! I have also already dropped several other new code datasets and rolled out QoL improvements for older ones. I will be dropping posts on those throughout the week.

new activity 4 days ago

nyuuzyou/joyreactor:[bot] Conversion to Parquet

View all activity

Organizations

nyuuzyou 's collections 5