Principal AI Engineer — Agentic Systems & AI Infrastructure

About Hashmark Labs

Hashmark Labs is a senior-led engineering consultancy building production-grade systems across AI, Web3 and FinTech for international clients.

We are looking for an exceptional Principal AI Engineer who can independently architect, engineer and deploy sophisticated AI systems, with deep expertise in agentic workflows, retrieval, evaluation and AI-native infrastructure.

This is not a prompt-engineering or prototype-only role. We need an experienced engineer who has personally shipped production AI systems and can own complex technical delivery from architecture to scale.

What You'll Build

Agentic AI Systems: Multi-agent architectures, graph-based orchestration, planning, routing, tool calling, memory, queues and human-in-the-loop workflows.

RAG & Enterprise Search: Hybrid retrieval, embeddings, vector databases, metadata-aware reranking, multi-tenant knowledge systems and access control.

LLM Evaluation & Observability: Evaluation datasets, quality metrics, tracing, model benchmarking, regression testing and reliable production monitoring.

Multimodal & Document Intelligence: AI pipelines processing PDFs, tables, images, voice and complex enterprise information with traceable outputs.

AI-Native Infrastructure: Scalable APIs, asynchronous services, distributed workloads, data pipelines, containerized/cloud deployments and cost-efficient inference.

End-to-End Technical Ownership: Translate complex client requirements into robust architectures, production implementations and measurable business outcomes.

What We're Looking For

Exceptional Python engineering with strong backend architecture and distributed-systems fundamentals.

Deep production experience with LLMs, AI agents, tool calling, structured outputs and agent orchestration.

Strong expertise in RAG, vector search, retrieval optimization and enterprise AI systems.

Practical experience with LLM evaluation, observability, reliability engineering and performance optimization.

Hands-on experience with FastAPI, PostgreSQL/pgvector, vector databases, Redis, Kafka, Docker, Kubernetes, AWS or GCP.

Preferred: Experience building enterprise SaaS, financial AI, underwriting or decision systems, voice agents, multimodal platforms, regulated workflows or AI + blockchain applications.

We value demonstrated engineering depth and systems ownership over arbitrary years of experience.

How We Work

Our engineering philosophy is evaluation-first and production-focused.

We prioritize measurable accuracy, grounded outputs, system reliability, data security, latency, cost efficiency and maintainable architectures.

You will collaborate directly with international product and engineering teams while retaining hands-on ownership of critical technical decisions.

Engagement & Compensation

Type: Contractual engagement

Location: Fully remote — worldwide

Commitment: Approximately 5–6 hours/day, 5 days/week

Schedule: Flexible working hours

Compensation: Negotiable based on experience, technical expertise and demonstrated production ownership

Performance-based revisions: Compensation may be reviewed and increased based on delivery and project performance, potentially including increases of 50% or more. Such revisions are not guaranteed.

International payments: USDC or INR arrangements may be discussed where applicable

Flexibility: Other professional engagements are permitted, subject to meeting agreed commitments

How to Apply

Apply through Wellfound and share your experience with 2–3 production AI systems you personally architected or materially engineered.

Briefly describe your technical ownership, architecture decisions, evaluation methods and production challenges.

Recruitment inquiries: [hidden]

We welcome experienced AI engineers and architects worldwide.