Engineering Lead - workflow Orchestration (Remote)
Recruit Square Verified employer
Actively hiring · Posted today · JEV26114AF4F3ECC2
Job description
We are seeking a seasoned Engineering Lead with deep expertise in workflow orchestration systems, stateful execution engines, and distributed task runtimes. You will architect the next generation of our DAG-based runtime, develop the infrastructure for agentic workflow composition, and drive execution excellence across engineering. This role combines hands-on systems design with technical leadership and team mentorship.Key Responsibilities1. Architect & Build the Swarm RuntimeDesign and implement a DAG-based orchestration engine using Temporal, Argo Workflows, or equivalent event-driven runtimes.Build a scalable primitive registry for tasks, operators, guards, and computational nodes.Architect a robust scheduler capable of handling event triggers, retries, backoffs, and distributed coordination.2. Develop the Workflow Composition LayerDefine and build a YAML/JSON-based DSL for describing agentic workflows, dependencies, and execution semantics.Create a schema-driven rules engine ensuring validations, model calls, parallelism, conditional branching, and approval gates are seamlessly integrated.3. Orchestration Logic & Runtime IntelligenceImplement orchestration logic that coordinates:Validation layersModel calls (LLMs, embedding engines, external APIs)Human-in-the-loop approval gatesStateful transitions and checkpointingEnsure deterministic execution, traceability, and safe rollback mechanisms.4. Workflow Certification & Automated TestingDefine certification standards for every workflow type, including:Functional correctnessLatency and concurrency thresholdsError-handling expectationsObservability and trace coverageBuild automated regression test suites validating workflow integrity before deployment.5. Engineering Leadership & DeliveryLead, mentor, and grow a team of backend and systems engineers.Drive sprint planning, reviews, engineering discipline, and roadmap execution.Own runtime delivery deadlines, cross-team coordination, and release quality.6. Observability, Reliability & Production ExcellenceInstrument the runtime with observability hooks: metrics, tracing, structured logs, and execution heatmaps.Build robust retry logic, distributed locks, idempotency guards, and failover strategies.Improve runtime stability, throughput, and scale characteristics.RequirementsMust-Have8+ years of backend engineering experience building high-scale systems.3+ years leading teams focused on workflows, automation, orchestration, or distributed runtimes.Deep understanding of:Stateful orchestration engines (Temporal, Step Functions, Argo, Airflow)Message queues, pub/sub systems, and event-driven patternsRetry logic, compensating transactions, and idempotent operationsDistributed tracing, observability pipelines, and health checksStrong background in concurrent programming, async task management, and execution models.Hands-on experience with at least one systems language or backend stack (Python, Go, Rust, Node).Nice-to-HaveExperience building workflow DSLs or schema-driven interpreters.Familiarity with LLM pipelines, agentic runtimes, or AI-driven workflow automation.Knowledge of Kubernetes-based runtime environments and workflow controllers.Experience with pluggable architecture design, sandboxing, or execution policies.What This Role OffersOwnership of the core execution engine powering Perceive Now’s intelligent automation platform.A high-impact leadership position shaping architectural strategy and engineering culture.The opportunity to solve cutting-edge problems at the intersection of orchestration, distributed systems, and AI.
Key skills
How to apply
- Sign in or create your free profile — takes about two minutes.
- Apply with one tap — your profile is shared directly with Recruit Square.
- Track your application and get updates by email and WhatsApp.