Data Development Engineer (Real-time Computing)

Location: China [Position Overview] 1. The company has established a relatively complete data warehouse and real-time data system with capabilities in data collection, computing, storage and services. With the growth of business scale and increased real-time application scenarios, we are looking to recruit two mid to senior-level data development engineers responsible for real-time data requirement delivery, task optimization and engineering governance. 2. This position focuses primarily on real-time data development, while also participating in offline data warehouse construction. You will focus on solving data latency, state expansion, resource consumption, data quality and consistency issues in the real-time data pipeline, driving the real-time data system toward greater stability, efficiency and standardization. 【岗位职责】 1. Responsible for analysis, solution design and development delivery of real-time data requirements, building real-time data processing pipelines based on technologies such as Flink and Kafka, providing data support for real-time analysis, monitoring, metrics and business decision-making scenarios. 2. Participate in real-time data warehouse construction and model design, complete data stratification, metric caliber, dimensional model and data service design and implementation based on business requirements. 3. Continuously optimize existing real-time tasks, addressing data latency, backpressure, Checkpoint exceptions, state expansion, unreasonable resource utilization and data inconsistency issues, improving task stability and computing efficiency. 4. Participate in data governance and engineering quality construction, including task dependency clarification, metadata management, data quality monitoring, alerting mechanisms, development standards and problem diagnosis system construction. 5. Participate in daily maintenance and issue handling of big data platforms and computing engines, coordinating with upstream and downstream teams to ensure stability, accuracy and timeliness of data pipelines. 6. Accumulate real-time development best practices and technical experience, improving the team's overall real-time R&D capability through technical sharing, code reviews and solution reviews. [Job Requirements] 1. Bachelor's degree or above in Computer Science, Software Engineering, Information Technology or related field, with 5 years or more of data development or big data R&D experience; flexibility may be granted for exceptionally capable candidates. 2. Possess complete data warehouse project experience, familiar with real-time data warehouse architecture, data stratification and data model design, able to transform business requirements into data models and technical solutions. 3、熟练掌握Flink，具有Flink SQL、DataStream API等实际项目经验，理解State、Checkpoint、Watermark、窗口、反压和容错机制。 4. Possess real-time task development and optimization experience, able to independently analyze and resolve issues such as task latency, high resource consumption, state exceptions, data skewing and data consistency. 5. Familiar with Kafka and real-time data pipelines, understanding the complete process of data collection, transmission, computing, storage and services. 6. Familiar with Hadoop, Hive, Spark and other big data technologies, with certain offline data development capabilities. 7. Proficient in SQL, capable of completing complex data processing and performance optimization; familiar with at least one development language among Python, Java or Scala. 8、熟悉MySQL、PostgreSQL、Elasticsearch等常用数据库或存储系统。 9. Possess good business understanding, communication and collaboration skills and written expression ability, strong sense of responsibility, able to proactively drive problem resolution and project implementation. [Bonus Qualifications] 1. Experience with large-scale Flink task development, performance tuning or cluster governance. 2. Experience with real-time data warehouse architecture optimization, real-time pipeline governance or technical upgrades. 3、熟悉Hologres、ClickHouse、Doris等实时OLAP技术。 4. Experience with complex business scenarios such as real-time metrics, real-time monitoring, real-time risk control or real-time recommendations. [The Candidates We Expect] 1. We hope that you can not only complete data requirement development, but also be willing to deeply understand real-time computing mechanisms, focusing on the stability, performance and engineering quality of data pipelines. 2. When facing complex problems, you can identify root causes from multiple angles including data, code, task configuration and running mechanisms, and consolidate solutions into reusable standards, tools or best practices.