🌍 Intelligence Analyst, Remote- Contract

Role Title: Intelligence Analyst Role Type: Contract Location: Remote Required Skills: - Analytic rigor - Source handling - Over-refusal detection - All-Source Intelligence Analysis In this role, you'll apply your expertise to help train next-generation AI systems. Your work will shape how models learn, reason, and perform through high-quality, real-world input. No prior experience in AI is required - your domain knowledge is what matters. Scope of Work: - Develop and map analytic cases to defined taxonomies across multiple evaluation areas, ensuring comprehensive coverage of relevant intelligence scenarios. - Author complex test cases designed to evaluate model performance on source handling, analytic rigor, and nuanced uncertainty expression. - Assess and grade AI-generated responses based on criteria such as reasoning, source use, recognition of limitations, and practical utility. - Identify and flag instances where legitimate analytic exploration is over-refused by the model, providing clear documentation and feedback. - Document edge cases and analytic disagreements, supporting continuous improvement of evaluation methodologies. - Collaborate with project leads to refine procedures for source evaluation and confidence-level assignment. - Contribute insights to help calibrate analytic standards and enhance the overall quality of AI system benchmarks. Preferred Qualifications: - Demonstrated all-source analytic experience or deep expertise in a specific analytic discipline, ideally with a civilian-IC background. - Strong proficiency in source evaluation, confidence-level assessment, and established analytic standards. - Experience in teaching, mentoring, or leading analytic reviews, with a record of fostering analytic rigor. - Exceptional written and verbal communication skills, with a focus on clarity and precision. - Ability to identify and articulate nuanced model limitations and over-refusal scenarios. - Familiarity with structured analytic techniques and intelligence tradecraft standards. - Previous participation in AI evaluation or model assessment initiatives is a plus. Originally posted on Himalayas