混元大模型漫爬数据策略算法工程师
Core
Design and optimize web crawling strategies for large model pre-training and AI search, covering multi-language and multi-modal data discovery, scheduling, quality filtering, and content understanding.
Role type
Senior IC machine-learning engineer (crawling strategy & data engineering)
Builds
Large-scale training datasets for LLMs and AI search systems
Domain
AI/ML, Data Engineering, Web Crawling
Deliverable
production ML models
Required skills
Python, C++, Machine Learning, NLP, LLMs, BERT, GPT, Hadoop, Spark, SQL, Data Warehousing
Preferred skills
Computer Science degree, Data Engineering experience
Technologies
Python, C++, Hadoop, Spark, BERT, GPT, LLMs, SQL
Responsibilities
Design global crawling strategies for resource discovery and scheduling; Build content filtering strategies using ML/NLP/LLM to identify low-quality or duplicate pages; Optimize URL and content deduplication strategies to improve crawling efficiency; Develop web content understanding and classification models to assist LLM training and AI search.
Seniority
Senior, hands-on IC
