Sr Engineer - Observability (Xico)

Sr Engineer – Observability

Key Responsibilities

• Architect and implement the Manager of Managers (MOM) tool that will be the "pane of glass" used by the Virtual Integrated Command Center operators.
• Analyze, design, program, debug, and modify observability tools and interfaces.
• Enrich and correlate telemetry from many data sources to isolate events indicating future or immediate IT availability issues.
• Interact with users to define system requirements and/or necessary modifications.
• Design and implement observability solutions utilizing industry‑standard tools such as Prometheus, Grafana, ELK Stack (Elasticsearch, Logstash, Kibana), Jaeger, and OpenTelemetry; implement distributed tracing to trace and visualize request flows across microservices architectures and use tracking data to identify performance bottlenecks and optimize system performance.
• Analyze system performance metrics, identify opportunities for optimization, and collaborate with development teams to implement performance improvements and ensure scalability.
• Participate in incident response activities, diagnose and resolve complex issues, conduct thorough post‑incident reviews to identify root causes and recommend preventive measures.
• Document observability best practices, standards, and procedures, and share knowledge through presentations, workshops, and documentation to foster a culture of continuous learning and improvement.
• Collaborate with cross‑functional teams including DevOps, SRE, and software engineering to drive observability initiatives and ensure alignment with organizational goals and objectives.

Qualifications

• Bachelor's or Master's degree in Computer Science, Information Technology, or related field.
• 2+ years of software engineering experience focused on observability, monitoring, and/or site reliability engineering.
• 1-2 years of experience with one or more of the following: APM, Monitoring/Alerting, New Relic, Dynatrace, AppDynamics, Zabbix, BigPanda, and ServiceNow.
• Proficiency in designing and implementing observability solutions using Prometheus, Grafana, ELK Stack, Jaeger, and OpenTelemetry.
• Strong understanding of distributed systems, microservices architectures, and cloud computing platforms (AWS, Azure, GCP).
• Experience with containerization technologies such as Docker and Kubernetes.
• Ideally 2+ years of development experience with programming languages such as C#, .NET, or JavaScript.
• Excellent analytical and problem‑solving skills with strong attention to detail.
• Effective communication and collaboration skills, with the ability to work across teams and influence stakeholders.
• Experience working in an Agile/Scrum environment is preferred.