[Remote] Lead Engineer – HPC Operations

Note: The job is a remote job and is open to candidates in USA. Core42 is a leader in AI-powered cloud and digital infrastructure, driving transformative technology solutions globally. They are seeking a highly skilled Lead Engineer – HPC Operations to oversee the daily operations and support of high-performance computing clusters designed to power large-scale AI and ML workloads, ensuring stable and high-performing infrastructure.

Responsibilities

• Oversee the daily operational management of HPC infrastructure, including compute, storage, networking, and scheduler components (e.g., Slurm, Kubernetes, etc.)
• Drive efforts to optimize the efficiency and performance of HPC systems, ensuring maximum resource utilization and minimizing downtime
• Serve as the primary technical escalation point for L2 support teams, ensuring rapid and effective resolution of incidents and service requests
• Continuously monitor system health, performance, and resource utilization using advanced monitoring tools (e.g., Prometheus, Grafana, DCGM)
• Manage user environments for AI/ML workloads, including container orchestration (e.g., Docker, Kubernetes) and workflow tools (e.g., MLflow, Kubeflow)
• Define and enforce job scheduling policies, priorities, and partitions within Slurm and/or Kubernetes environments to ensure resource fairness, efficiency, and workload optimization
• Lead root cause analysis (RCA) of operational issues, contributing to post-mortem documentation and driving continuous improvement initiatives
• Provide mentorship and technical guidance to junior engineers, fostering skills development and knowledge sharing across teams. Participate in on-call rotation as necessary
• Ensure adherence to security and operational policies, assisting in audits and maintaining documentation for change and incident management processes

Skills

• Bachelor's or Master's degree in Computer Science, Engineering, or a related technical field
• Minimum of 8 years of experience in HPC operations, systems engineering, or DevOps roles, with at least 2 years in a leadership or ownership capacity
• Advanced expertise in configuring, optimizing, and maintaining complex HPC environments, including hardware, software, and storage systems
• Hands-on experience managing Slurm clusters and/or Kubernetes-based environments for AI/ML workloads
• In-depth knowledge of GPU resource management, workload schedulers, and performance tuning for AI/ML workloads
• Proficiency with monitoring and observability frameworks such as Prometheus, Grafana, and DCGM
• Strong scripting and automation skills, including Python, Bash, Ansible, and Terraform
• Solid understanding of Linux (RHEL/CentOS/Ubuntu), networking technologies (RDMA, InfiniBand, RoCE), and storage solutions (NFS, Lustre, Ceph)

Benefits

• Bonus and benefits on top

Company Overview