Senior Software Engineer – ModelVault

ModelVault is building the world’s first fully-integrated, on-prem AI appliance — a plug-and-play AI server preloaded with software, models, and tools that let businesses (and eventually consumers) run powerful AI models locally, privately, and securely. We're looking for a Senior Software Engineer to help us build the core software stack that powers our on-prem systems — from model integration and system orchestration to user interfaces and remote updates. You’ll be a key technical partner in defining our architecture, tools, and technical culture from day one. 🚀 What You’ll Do - Design and build the software stack that ships with ModelVault’s on-prem AI systems — everything from containerized inference engines to local dashboards and command-line tools. - Set up and manage integration with modern LLMs and vision models (e.g. LLaMA, Mistral, SDXL), including support for fine-tuning and local deployment. - Build and maintain user-facing tooling: CLIs, lightweight GUIs, and APIs that make the device intuitive for developers and AI teams. - Make architecture decisions for system orchestration, hardware interfacing, remote update pipelines, and telemetry — all optimized for stability and security. - Help set up GPU drivers, OS configuration (e.g., Ubuntu, NVIDIA stack), and optimized inference runtimes as needed. - Collaborate with the founding team to test and iterate with early business customers. - Contribute to hiring and scaling the engineering team over the next 12 months. 🧠 You Might Be a Fit If You - Have 5+ years of software engineering experience, preferably at early-stage startups or in fast-paced environments. - Are comfortable working across the stack — from OS-level setup and deployment to user-facing tools. - Have experience with containerization, system packaging, deployment tooling, and managing model runtimes. - Have worked with, or integrated, LLMs and ML models in real-world systems (e.g., inference, fine-tuning, embeddings). - Can independently drive architecture decisions and own large pieces of product infrastructure. - Thrive in ambiguity, learn fast, and are passionate about building great tools from scratch. ⭐️ Bonus Points - Experience with GPU computing, NVIDIA drivers, CUDA, or model acceleration tools. - Familiarity with ML tooling: vLLM, Ollama, Triton Inference Server, LangChain, etc. - Comfort with Linux system internals, remote access tooling (e.g., SSH, Ansible), or auto-update agents. - Previous experience with AI developer tooling, embedded systems, or secure local apps. 🌍 Why Join ModelVault? - Help define a new category: secure, local AI for everyone — starting with businesses, expanding to the world. - Work on real, shipped hardware + software that users will touch. - Be part of the founding engineering team with high autonomy and deep ownership. - Fully remote, async-friendly, with strong product velocity and clear vision. Originally posted on Himalayas