Responsibilities:

• Keep all the nodes and network running smoothly.
• Handle and fix any and all problems that may arise.
• Determine the architecture for overall system operation and new system projects in collaboration with technical management.
• Maintain communication with b2b customers regarding system operations and development issues, clarification of customer needs, and coordination with DevOps teams for system enhancements and innovations.
• Keep all test automation and infrastructure running smoothly.
• Monitor log files, performance issues, and the production environment for mission-critical systems.
• Participate in an on-call rotation for Linux support issues (1 week in every 3).
• Follow standard operating procedures to ensure the security posture of the environment.
• Engage in continuous learning to stay abreast on new and emerging technologies that relate to the Linux environment.
• Ensure compliance with established standards, policies, and configuration guidelines.
• Maintain a cooperative working relationship with those contacted during the course of the work day.

Our tech stack: 

• 20+ AWS accounts, managed by Terraform/Terragrunt.
• AWS services in use – IAM, S3, EC2, ECR, NLB, CloudWatch, AuditTrail.
• ~800 VMs/LXD containers running on Ubuntu LTS across public/private clouds.
• Self-managed, big data databases such as Apache HBase (for OLTP) and Starrocks (for OLAP) – total size of databases – hundreds of TBs and processing hundreds of thousands operations per second. We have to run on every infrastructure so Databases as a Service is not an option for us, we have to be able to manage it ourselves. The database tier is not containerized.
• Containerized application tier- currently based on a custom orchestration scheme using Ansible, Docker Compose, and Haproxy. Currently, we don’t use k8s for the application tier.
• Kafka clusters for distributed streaming, messaging, and asynchronous jobs needs.
• Redis clusters in Sentinel mode.
• Monitoring stack based on Alloy; Telegraf; Victoria’s vmagent, VictoriaMetrics, VictoriaLogs, Grafana, AlertManager.
• CI/CD using GitHub Actions.
• Infrastructure as a code using Terraform/Terragrunt, Ansible and git.

Requirements:

• 5+ years of relevant DevOps, Infrastructure, Platform Engineering, or Site Reliability Engineering experience.
• Strong Ansible – it’s the most used tool in our daily DevOps activities.
• Solid Linux experience – all the essential services.
• Terraform managed AWS experience.
• Experience with managing (not using it as Database-as-a-Service) big-scale databases, preferably Hbase, but others count too such as Cassandra, Yugabyte, TiDB, or the sharded forms of Mongo, MySQL, etc. in multi TB scale.
• Container-based deployments.
• CI/CD pipelines experience.
• Bash scripting.
• Experience with planning and building complex infrastructure projects from scratch.
• A self-organized and motivated person, who doesn’t need constant supervision – the team’s culture is not based on waiting for a Jira ticket to be opened from someone else, instead, we’re continuously looking for improvements not driven by an external trigger.
• Experience with serving big scale web, api platforms.
• Experience with some monitoring and centralized logging platforms – such as Grafana, Netdata, Loki, Victoria Metrics, and Prometheus.
• The person should be able to participate in an on-call rotation – 1 week in every 3.

Nice-to-have requirements:

• Hadoop stack – the HDFS filesystem and HBase database
• Redis clusters
• Kafka clusters
• Zookeeper 
• Experience in the administration of Java-based applications
• Experience with programming languages – such as Python, Go, for Ops needs

Please note that you don’t need experience with every technology in our stack, but you should be comfortable learning and operating complex infrastructure systems and willing to grow and learn the rest with us.