# Abhishek Divekar > Abhishek Divekar is an AI researcher at Amazon working on LLM evaluation (LLM-as-a-Judge), synthetic data generation, red-teaming and guarding LLMs, and multimodal agents for science. First-author papers at NeurIPS 2026, CIKM 2026, ACL 2026, AAAI 2026 and EMNLP 2024. Chair of the IOAI² AI Models Track and member of the International Scientific Committee of the International Olympiad in Artificial Intelligence (IOAI 2026); Team Leader of Team India at IOAI 2025 (3 Gold, 2 Silver, 1 Bronze, 1 Honorable Mention). Abhishek Divekar: Senior Researcher, Amazon. I research LLM evaluation, synthetic data, red-teaming LLMs and multimodal agents for science, with first-author papers at NeurIPS, ACL, EMNLP, AAAI, and CIKM. I collaborate with researchers at UT Austin, NYU and MBZUAI. At Amazon, I turn that research into products, including leading red-teaming and evals for the India launch of Amazon Rufus, [used by 100M+ customers ↗](https://www.aboutamazon.in/news/retail/rufus-ai-shopping-assistant-launch-in-india). My scientific and mathematical contributions have added over USD 550 million to Amazon's revenue between 2021-2026. I have deep industry experience working across several areas of e-commerce science; this includes building RL environments for LLM post-training, Search and Ads Ranking, Online advertising auction simulators, LLM-as-a-Judge, user-facing chatbot applications (Rufus), and LLM red-teaming & guardrails. I have shipped 650,000 lines of code into production (~500k was written pre-LLMs), including distributed language model training. My current favorite language model is Claude Opus 5.5 (high effort). I volunteer on the Scientific Committee of the International Olympiad in AI (IOAI 2026 & 2027), where I chaired the inaugural [IOAI²: AI Models Track ↗](https://ioai-official.org/ai-models-track/), a first-of-its-kind competition where I recruited 14 AI labs from industry and academia to solve AI Olympiad problems using fully autonomous agentic systems. Previously, I served as India's Team Lead (AI Olympiad Coach) at IOAI 2025, leading a team of 8 students to win 3 Gold, 2 Silver, 1 Bronze medals, and 1 Honorable Mention. I hold an M.S. in Computer Science from UT Austin, where I was advised by [Greg Durrett ↗](https://gregdurrett.github.io/). Links: [Google Scholar](https://scholar.google.com/citations?user=1Sf2oM4AAAAJ), [GitHub](https://github.com/adivekar-utexas), [LinkedIn](https://www.linkedin.com/in/ardivekar), [CV (PDF)](https://adivekar.com/files/Abhishek-Divekar-CV.pdf). Site map: [Home](https://adivekar.com/) · [Writing](https://adivekar.com/writing/) · [Speaking](https://adivekar.com/speaking/) · [CV (PDF)](https://adivekar.com/files/Abhishek-Divekar-CV.pdf) ## News - Dec 2026 [paper]: VESTA accepted to NeurIPS 2026 (main track)! [Read the paper](https://arxiv.org/abs/2606.00384) - Nov 2026 [paper]: TrojanShot (with Vijit Malik) published at CIKM 2026, Rome! [Read the paper](https://dl.acm.org/doi/10.1145/3799682.3841000) - Nov 2026 [AI Olympiad]: Re-elected as a core member of the IOAI 2027 Singapore International Scientific Committee, and of the IOAI 2027 AI Models Track Core Committee. [See my AI Olympiad work](https://adivekar.com/#ioai) - Sep 2026 [AI Olympiad]: Released the IOAI² 2026 review, co-authored with IOAI co-founder Elena Marinova. [Read the report](https://ioai-official.org/ai-models-track/ioai%C2%B2-2026-report/) - Aug 2026 [AI Olympiad]: Chaired the inaugural IOAI² AI Models Track at IOAI 2026, Astana, Kazakhstan, where 14 AI organizations from 5 countries participated to benchmark fully autonomous AI systems at solving IOAI 2026 Tasks. [Read the report](https://ioai-official.org/ai-models-track/ioai%C2%B2-2026-report/) - Aug 2026 [AI Olympiad]: The International Jury's final report at the IOAI 2026 closing ceremony presented the AI Models Track. [Read the report](https://ioai-official.org/ai-models-track/ioai%C2%B2-2026-report/) - Aug 2026 [AI Olympiad]: Travelled to Kazakhstan to serve on the IOAI 2026 Jury. [See the Jury](https://ioai-official.org/republic-of-kazakhstan/international-jury/) - Jul 2026 [paper]: When Gradients Collide (with Parth Darshan) accepted to the ACL 2026 CustomNLP4U workshop! [Read the paper](https://arxiv.org/abs/2605.26046) - Jun 2026 [paper]: Statistically Reliable LLM-Based Ranking Evaluation via Prediction-Powered Inference, an extended abstract of PRECISE, accepted to the ACL 2026 GEM workshop! [Read the paper](https://arxiv.org/abs/2606.05308) - Jan 2026 [award]: PRECISE won the IAAI 2026 Deployed Application Award in the AAAI 2026 Innovative Applications of AI (IAAI) track, Singapore! [Read the paper](https://arxiv.org/abs/2601.18777) - Jan 2026 [talk]: Presented PRECISE in an oral session at AAAI 2026 (Innovative Applications of AI track, 19.2% acceptance rate), Singapore. [Read the paper](https://arxiv.org/abs/2601.18777) - Dec 2025 [AI Olympiad]: Joined the IOAI International Scientific Committee for IOAI 2026. [See my AI Olympiad work](https://adivekar.com/#ioai) - Oct 2025 [paper]: TrojanShot selected as an oral at the Amazon Machine Learning Conference 2025! [Read the paper](https://adivekar.com/#trojanshot-amlc-2025) - Aug 2025 [AI Olympiad]: Led Team India at IOAI 2025, Beijing, China: 3 Gold, 2 Silver, 1 Bronze, and 1 Honorable Mention. [See the results](https://ioai-official.org/china-2025/results-2025/) - Aug 2025 [award]: Named one of the Top 7 GenAI Scientists in India by Analytics Vidhya at DataHack Summit 2025. - Aug 2025 [award]: Also named one of the Top 7 GenAI Community Contributors at the same DataHack Summit 2025. - Aug 2025 [talk]: Invited talk on synthetic data generation at DataHack Summit 2025. [See the talk](https://adivekar.com/speaking/#dhs-2025-synthetic-data) - May 2025 [AI Olympiad]: Served on the Technical Committee of the ACM India AI Olympiad (INAIO 2025), Hyderabad, India. - Feb 2025 [Collaboration]: Joined the NSF-Simons AI Institute for Cosmic Origins (CosmicAI) as a Research Collaborator. - Nov 2024 [paper]: SynthesizRR and CorrSynth accepted to EMNLP 2024 (main conference)! [Read the papers](https://adivekar.com/#publications) - Oct 2024 [paper]: SynthesizRR selected as an oral at the Amazon Machine Learning Conference 2024! [Read the paper](https://adivekar.com/#synthesizrr-amlc-2024) - Oct 2024 [Product launch]: Launched PRECISE in GlowRoad search, measuring search relevance with LLM judgments and few human labels. [See the launch](https://adivekar.com/#shipped) - Aug 2024 [education]: My M.S. thesis, "Diverse dataset synthesis using retrieval augmented language models", published by The University of Texas at Austin. [Read the thesis](https://repositories.lib.utexas.edu/items/ddb94d0e-928d-406f-a49f-632995fd4b03) - May 2024 [education]: Completed my M.S. in Computer Science at UT Austin, advised by Greg Durrett. - Oct 2022 [talk]: Presented Flexible AutoML at AIMLSys 2022. [See the slides](https://adivekar.com/speaking/#litmus-aimlsys-2022-talk) - Jun 2022 [talk]: Taught supervised learning at the Amazon ML Summer School to about 3,000 Indian undergraduates. [See the lecture](https://adivekar.com/speaking/#ml-summer-school-2022-lecture) - Oct 2021 [talk]: Presented DRiP, AutoML for cost-constrained product classification, at Amazon Research Days 2021. [See the slides](https://adivekar.com/speaking/#drip-ard-2021-talk) - Sep 2021 [paper]: DRiP and the CPP multimodal AutoML benchmark accepted at the Amazon Machine Learning Conference 2021! [Read the papers](https://adivekar.com/#amlc) - Aug 2020 [education]: Started my M.S. in Computer Science (thesis track) at UT Austin, part-time alongside Amazon. - Oct 2019 [Full-time]: Moved from software engineering to Applied Science at Amazon Central Machine Learning. [Read how](https://adivekar.com/writing/engineer-to-ml-scientist/) - Oct 2018 [paper]: First-author paper on intrusion-detection benchmarks presented as an oral at IEEE ICCCS 2018! [Read the paper](https://arxiv.org/abs/1811.05372) - Aug 2017 [award]: Won the Amazon Machine Learning Challenge in Chennai, out of 300 participants. - Jul 2017 [Full-time]: Joined Amazon as a Software Development Engineer (Digital Payments). - Apr 2017 [education]: Graduated from VJTI, Mumbai with a B.Tech. in Information Technology. ## Career ### Full-time - Dec 2021 to now: Senior Researcher, Amazon. - Oct 2019 to Nov 2021: Applied Scientist II, India Machine Learning, Amazon. - Jul 2017 to Sep 2019: Software Development Engineer, Digital Payments, Amazon. ### Part-time - Feb 2025 to now: Research Collaborator, NSF-Simons AI Institute for Cosmic Origins (CosmicAI). LLM and vision-language agents for astronomy, with UT Austin, NYU and UCLA. ### Academic service - Nov 2026 to now: Member, International Scientific Committee (IOAI 2027, upcoming), International Olympiad in Artificial Intelligence. - Dec 2025 to Sep 2026: Member of IOAI Scientific Committee, and Chair of IOAI²: AI Models Track (IOAI 2026), International Olympiad in Artificial Intelligence. Duties: Chair of the IOAI² AI Models Track; member of the IOAI Jury. - May 2025 to Aug 2025: Team Lead, Team India (IOAI 2025), International Olympiad in Artificial Intelligence. Selected, mentored and coached the 8 olympians: 3 Gold, 2 Silver, 1 Bronze, 1 Honorable Mention. - Feb 2025 to May 2025: Technical Committee, ACM India AI Olympiad (INAIO 2025), ACM India AI Olympiad. Set Stage-2 problems and judged the national selection rounds. ### Internships - May 2016 to Jul 2016: Software Development Intern, Barclays. Built a prototype that cut foreign-exchange settlement time from T+2 days to T+120 seconds, using Ripple blockchain and Node.js. ### Education - Aug 2020 to May 2024: M.S. Computer Science (thesis track), The University of Texas at Austin. Advised by Greg Durrett. Completed part-time while working full-time at Amazon. My thesis was published at EMNLP 2024 as a first-author main conference paper (SynthesizRR). [Read the thesis](https://repositories.lib.utexas.edu/items/ddb94d0e-928d-406f-a49f-632995fd4b03) - Jul 2013 to Apr 2017: B.Tech. Information Technology, Veermata Jijabai Technological Institute (VJTI), Mumbai. Bachelor's thesis on machine learning for anomaly-based network intrusion detection, advised by Prof. Mahesh Shirole; I was a Research Assistant in his lab (Apr to Jun 2017) and published the work as a first-author paper at IEEE ICCCS 2018. [Read the paper](https://arxiv.org/abs/1811.05372) Opinions on this site are my own and do not reflect the views of my employer. ## Publications \* = equal contribution. [Visit my Google Scholar profile](https://scholar.google.com/citations?user=1Sf2oM4AAAAJ) ### VESTA: Visual Exploration with Statistical Tool Agents - Venue: NeurIPS 2026 (Main track), Poster, 25.7% Accept Rate - Authors: William Rudman*, Abhishek Divekar*, Kanishk Jain*, Sebastian Joseph, Stella S. R. Offner, Matthew Lease, Kyle Mahowald, Greg Durrett, and Junyi Jessy Li - Paper: https://arxiv.org/abs/2606.00384 - Code: https://github.com/wrudman/VESTA - TL;DR: Vision-language agents that build their own diagnostic tools while fitting statistical models to scientific data, including real astronomy tasks. - Abstract: Fitting quantitative models to data is a central step in scientific workflows, yet it remains one of the least automated. Recent agent-based systems leverage language and vision-language models (VLMs) to iteratively propose and refine statistical models, but these systems struggle on more challenging modeling tasks. To address these limitations, we introduce VESTA (Visual Exploration with Statistical Tool Agents), a framework that equips VLMs with a dynamically growing exploration toolkit to guide model refinement through data transformations, hypothesis-driven visualizations, and robust statistical tests. Unlike prior systems that rely on iterative critique alone, VESTA actively explores data before and during refinement by selecting or creating diagnostic tools, which accumulate in the model's context and can be reused later. We evaluate VESTA against established baselines in three toolkit configurations: no tools, static expert-written tools, and dynamic model-written tools. To support this evaluation, we introduce DAWN (Dataset for Automated Workflows and Numerical Modeling), a benchmark targeting distribution fitting and time series modeling with varying difficulty tiers, and culminating in real-world astronomy tasks including modeling initial mass functions and gravitational-wave chirp signals. We find that VESTA's dynamic tool creation outperforms prior agentic pipelines, with the largest gains on complex and domain-specific tasks. We further show that dynamically generated tools are substantially more sophisticated than those produced by existing visual tool-creation systems, covering more diagnostic categories per function and strongly preferring visual outputs that the VLM critic can reason over directly. - BibTeX: ``` @inproceedings{rudman2026vesta, title = {{VESTA}: Visual Exploration with Statistical Tool Agents}, author = {William Rudman and Abhishek Divekar and Kanishk Jain and Sebastian Joseph and Stella S. R. Offner and Matthew Lease and Kyle Mahowald and Greg Durrett and Junyi Jessy Li}, booktitle = {The Fortieth Annual Conference on Neural Information Processing Systems}, year = {2026}, url = {https://openreview.net/forum?id=akBqLVEMrj}, abstract = {Fitting quantitative models to data is a central step in scientific workflows, yet it remains one of the least automated. Recent agent-based systems leverage language and vision-language models (VLMs) to propose statistical models, but these systems struggle on challenging modeling tasks. To address these limitations, we introduce VESTA (Visual Exploration with Statistical Tool Agents), a framework that equips VLMs with a dynamically growing toolkit to guide model refinement through data transformations, hypothesis-driven visualizations, and statistical tests. Unlike prior systems that rely on iterative critique alone, VESTA explores data before and during refinement by selecting or creating reusable diagnostic tools. We evaluate VESTA using three toolkit configurations: no tools, static expert-written tools, and dynamic model-written tools. To support this evaluation, we introduce DAWN (Dataset for Automated Workflows and Numerical Modeling), a benchmark targeting distribution fitting and time series modeling with varying difficulty tiers that culminates in real-world astronomy tasks including modeling initial mass functions and gravitational-wave chirp signals. VESTA's dynamic tool creation outperforms prior agentic pipelines for statistical analysis and visual tool creation, with the largest gains on complex and domain-specific tasks. Against strong agentic coding harnesses (Claude Code, Codex, and OpenCode), VESTA is better on all time series tasks and on complex distribution fitting tasks, though the harnesses achieve better fits on simpler distributions. Further, dynamically generated tools are substantially more sophisticated than those produced by existing visual tool-creation systems, covering more diagnostic categories per function and strongly preferring visual outputs that the VESTA VLM critic can reason over directly.} } ``` ### TrojanShot: Reinforced Demonstrations Yield Sample-Efficient Jailbreaks and Guardrails for LLMs - Venue: CIKM 2026 (Research track), Poster, 26.5% Accept Rate - Authors: Abhishek Divekar* and Vijit Malik* - Paper: https://dl.acm.org/doi/10.1145/3799682.3841000 - TL;DR: An RL-trained generator writes one in-context example that jailbreaks an LLM (80.6% attack success on Mistral-7B) or, flipped, guards it, cutting attack success by 48.9 points. - Abstract: LLM-based applications increasingly rely on models served through third-party providers, where the model's own alignment is often the sole defense against misuse. In this regime, developers must implement both red-teaming and guardrails themselves, yet existing in-context methods address these objectives in isolation and do not refine demonstrations against the target LLM's responses. We propose TrojanShot, a unified framework that post-trains a demonstration generator via reinforcement learning to produce potent in-context examples for both attack (Trojan shots) and defense (Guardian shots), operating entirely at the prompt layer of the target LLM. On AdvBench, a single Trojan shot reaches 80.6% ASR on Mistral-7B, and averaged across four datasets TrojanShot exceeds 256-shot Many-Shot Jailbreaking by 14.6 p.p. while preserving human-readable prompts; ablations show RL post-training is the dominant factor in demonstration quality, with the untrained generator already surpassing 1-shot ICA. We demonstrate strong transferability, with a trojan-attacker trained on Mistral-7B reaching 81.2% ASR on Llama-3-70B, exceeding the 69.9% of an attacker trained on Llama-3-8B. Guardian shots reduce TrojanShot's own ASR by 48.9 p.p. with helpfulness within 1.1 p.p. of the undefended model. Our results hold for models whose intrinsic alignment is the primary defense. - BibTeX: ``` @inproceedings{10.1145/3799682.3841000, author = {Divekar, Abhishek and Malik, Vijit}, title = {TrojanShot: Reinforced Demonstrations Yield Sample-Efficient Jailbreaks and Guardrails for LLMs}, year = {2026}, isbn = {9798400725395}, publisher = {Association for Computing Machinery}, address = {New York, NY, USA}, url = {https://doi.org/10.1145/3799682.3841000}, doi = {10.1145/3799682.3841000}, abstract = {LLM-based applications increasingly rely on models served through third-party providers, where the model's own alignment is often the sole defense against misuse. In this regime, developers must implement both red-teaming and guardrails themselves, yet existing in-context methods address these objectives in isolation and do not refine demonstrations against the target LLM's responses. We propose TrojanShot, a unified framework that post-trains a demonstration generator via reinforcement learning to produce potent in-context examples for both attack (Trojan shots) and defense (Guardian shots), operating entirely at the prompt layer of the target LLM. On AdvBench, a single Trojan shot reaches 80.6% ASR on Mistral-7B, and averaged across four datasets TrojanShot exceeds 256-shot Many-Shot Jailbreaking by 14.6 p.p. while preserving human-readable prompts; ablations show RL post-training is the dominant factor in demonstration quality, with the untrained generator already surpassing 1-shot ICA. We demonstrate strong transferability, with a trojan-attacker trained on Mistral-7B reaching 81.2% ASR on Llama-3-70B, exceeding the 69.9% of an attacker trained on Llama-3-8B. Guardian shots reduce TrojanShot's own ASR by 48.9 p.p. with helpfulness within 1.1 p.p. of the undefended model. Our results hold for models whose intrinsic alignment is the primary defense. Warning: this paper contains harmful model responses for academic reference.}, booktitle = {Proceedings of the 35th ACM International Conference on Information and Knowledge Management}, pages = {1012-1023}, numpages = {12}, keywords = {large language models, reinforcement learning, in-context learning, llm safety, red-teaming, defense, trojanshot, guardianshot}, location = {Rome, Italy}, series = {CIKM '26} } ``` ### When Gradients Collide: Failure Modes of Multi-Objective Prompt Optimization for LLM Judges - Venue: ACL 2026 (CustomNLP4U Workshop), Poster - Authors: Parth Darshan* and Abhishek Divekar* - Paper: https://arxiv.org/abs/2605.26046 - Code: https://github.com/adivekar-utexas/when-gradients-collide - Project: https://textgrad-failure-modes.github.io/ - TL;DR: Two failure modes of multi-objective prompt optimization for LLM judges: gradient dilution, a 59% specificity drop under joint critique, and instruction interference, a 5.3% Spearman loss. - Abstract: LLM-as-a-judge pipelines increasingly rely on automatic prompt optimization, where an outer LLM reads textual feedback ("gradients") about an inner judge and rewrites the judge prompt. When the judge scores multiple quality dimensions at once, the optimizer must reconcile several per-task feedback signals into a single revision. We study this multi-objective prompt optimization problem and identify two recurring failure modes. Gradient dilution: concatenating feedback from several tasks into one textual gradient makes the consuming LLM unable to tell which advice applies to which task, dropping gradient task-focus by 59% (9.0 to 3.7 on a 1-to-10 scale). Instruction interference: per-task instructions optimized in isolation interfere when stitched into a single multi-task prompt, degrading test-set Spearman from 0.305 to 0.220. We characterize both failures across four decomposition modes (SSS, SSC, SCC, CCC) on SummEval and BRIGHTER, using TextGrad and OPRO. - BibTeX: ``` @inproceedings{darshan-divekar-2026-gradients, title = {When Gradients Collide: Failure Modes of Multi-Objective Prompt Optimization for LLM Judges}, author = {Darshan, Parth and Divekar, Abhishek}, editor = {Mysore, Sheshera and Kumar, Sachin and Balachandran, Vidhisha and Hayati, Shirley Anugrah and Brahman, Faeze and Moussa, Hanane Nour and Salemi, Alireza}, booktitle = {Proceedings of the Second Workshop on Customizable NLP: Progress and Challenges in Customizing NLP for a Domain, Application, Group, or Individual (CustomNLP4U)}, month = jul, year = {2026}, address = {San Diego, California, USA}, publisher = {Association for Computational Linguistics}, url = {https://aclanthology.org/2026.customnlp4u-1.21/}, doi = {10.18653/v1/2026.customnlp4u-1.21}, pages = {216-228}, ISBN = {979-8-89176-396-8}, abstract = {Customizing an LLM judge to a specific task or domain often involves optimizing its prompt across multiple evaluation criteria simultaneously. Textual gradient methods automate this for a single judge criterion, however they produce natural-language critiques, not numerical vectors. Thus, the conflict-resolution toolkit of multi-task learning (PCGrad, MGDA) doesn't apply to the multi-objective textual gradient setting. We test five decomposition modes of textual gradient optimizers by varying how much cross-task information the loss, gradient and optimizer LLMs share. In 6 of 10 configurations on SummEval, we observe that optimization never improves over the initial prompt. Gradient specificity drops by 59% (from 9.0 to 3.7) when the gradient LLM processes multiple criteria jointly. Separately, we observe that naively combining per-task instructions into a single prompt degrades Spearman's rho by -5.3%. These results identify two separable failure modes: optimization-time gradient dilution and inference-time instruction interference, which together constrain the design space for multi-objective judge customization using textual feedback.} } ``` ### Statistically Reliable LLM-Based Ranking Evaluation via Prediction-Powered Inference - Venue: ACL 2026 (GEM Workshop), Oral - Authors: Abhishek Divekar - Paper: https://arxiv.org/abs/2606.05308 - TL;DR: Extended abstract of PRECISE: 30 human labels plus LLM judgments cut the standard error of Precision@4 by 21%, and in production the method picked the best of three variants, confirmed by A/B test (+407 bps daily sales). - Abstract: With PRECISE, we extended Prediction-Powered Inference to produce bias-corrected estimates of ranking evaluation metrics by combining a small human-labeled set with a large LLM-judged set. PPI is provably unbiased regardless of the LLM judge's error profile. We make it applicable to hierarchical metrics like Precision@K, where annotations are per-document but the metric is per-query, by reducing the output-space computation from O(2^|C|) to O(2^K). On the ESCI benchmark, augmenting 30 human annotations with Claude 3 Sonnet judgments reduces the standard error of Precision@4 estimates from 4.45 to 3.50 (a 21% relative reduction). In a production system, our framework correctly identified the best of three system variants from 100 human labels and 2 hours of domain-expert annotation; A/B testing confirmed this ranking with +407 bps in daily sales. - BibTeX: ``` @misc{divekar2026statisticallyreliablellmbasedranking, title = {Statistically Reliable LLM-Based Ranking Evaluation via Prediction-Powered Inference}, author = {Abhishek Divekar}, year = {2026}, eprint = {2606.05308}, archivePrefix = {arXiv}, primaryClass = {cs.LG}, url = {https://arxiv.org/abs/2606.05308}, abstract = {With PRECISE, we extended Prediction-Powered Inference to produce bias-corrected estimates of ranking evaluation metrics by combining a small human-labeled set with a large LLM-judged set. PPI is provably unbiased regardless of the LLM judge's error profile. We make it applicable to hierarchical metrics like Precision@K, where annotations are per-document but the metric is per-query, by reducing the output-space computation from O(2^|C|) to O(2^K). On the ESCI benchmark, augmenting 30 human annotations with Claude 3 Sonnet judgments reduces the standard error of Precision@4 estimates from 4.45 to 3.50 (a 21% relative reduction). In a production system, our framework correctly identified the best of three system variants from 100 human labels and 2 hours of domain-expert annotation; A/B testing confirmed this ranking with +407 bps in daily sales.} } ``` ### PRECISE: Reducing the Bias of LLM Evaluations using Prediction-Powered Ranking Estimation - Venue: AAAI 2026 (Industry track), Oral, 19.2% Accept Rate - Authors: Abhishek Divekar and Anirban Majumder - Paper: https://arxiv.org/abs/2601.18777 - TL;DR: Estimates search-quality metrics from about 100 human-labelled queries plus LLM judgments, correcting for the LLM judge's bias. Built for Amazon search. - Abstract: Evaluating the quality of search systems traditionally requires a significant number of human relevance annotations. In recent times, several systems have explored the usage of Large Language Models (LLMs) as automated judges for this task while their inherent biases prevent direct use for metric estimation. We present a statistical framework extending Prediction-Powered Inference (PPI) that combines minimal human annotations with LLM judgments to produce reliable estimates of metrics which require sub-instance annotations. Our method requires as few as 100 human-annotated queries and 10,000 unlabeled examples, reducing annotation requirements significantly compared to traditional approaches. We formulate our proposed framework (PRECISE) for inference of relevance uplift for an LLM-based query reformulation application, extending PPI to sub-instance annotations at the query-document level. By reformulating the metric-integration space, we reduced the computational complexity from O(2^N) to O(2^K), where N is the corpus size (in order of millions). Detailed experiments across prominent retrieval datasets demonstrate that our method reduces the variance of estimates for the business-critical Precision@K metric, while effectively correcting for LLM bias in low-resource settings. - BibTeX: ``` @article{Divekar_Majumder_2026, title = {PRECISE: Reducing the Bias of LLM Evaluations Using Prediction-Powered Ranking Estimation}, volume = {40}, url = {https://ojs.aaai.org/index.php/AAAI/article/view/41427}, DOI = {10.1609/aaai.v40i47.41427}, abstract = {Evaluating the quality of search systems traditionally requires a significant number of human relevance annotations. In recent times, several systems have explored the usage of Large Language Models (LLMs) as automated judges for this task while their inherent biases prevent direct use for metric estimation. We present a statistical framework extending Prediction-Powered Inference (PPI) that combines minimal human annotations with LLM judgments to produce reliable estimates of metrics which require sub-instance annotations. Our method requires as few as 100 human-annotated queries and 10,000 unlabeled examples, reducing annotation requirements significantly compared to traditional approaches. We formulate our proposed framework (PRECISE) for inference of relevance uplift for an LLM-based query reformulation application, extending PPI to sub-instance annotations at the query-document level. By reformulating the metric-integration space, we reduced the computational complexity from O(2^|C|) to O(2^K), where |C| represents corpus size (in order of millions). Detailed experiments across prominent retrieval datasets demonstrate that our method reduces the variance of estimates for the business-critical Precision@K metric, while effectively correcting for LLM bias in low-resource settings.}, number = {47}, journal = {Proceedings of the AAAI Conference on Artificial Intelligence}, author = {Divekar, Abhishek and Majumder, Anirban}, year = {2026}, month = {Mar.}, pages = {39929-39938} } ``` ### SynthesizRR: Generating Diverse Datasets with Retrieval Augmentation - Venue: EMNLP 2024 (Main), Poster, 20.8% Accept Rate - Authors: Abhishek Divekar and Greg Durrett - Paper: https://arxiv.org/abs/2405.10040 - Code: https://github.com/amazon-science/synthesizrr - TL;DR: Retrieval augmentation makes LLM-generated training data more diverse and closer to human-written text. - Abstract: It is often desirable to distill the capabilities of large language models (LLMs) into smaller student models due to compute and memory constraints. One way to do this for classification tasks is via dataset synthesis, which can be accomplished by generating examples of each label from the LLM. Prior approaches to synthesis use few-shot prompting, which relies on the LLM's parametric knowledge to generate usable examples. However, this leads to issues of repetition, bias towards popular entities, and stylistic differences from human text. In this work, we propose Synthesize by Retrieval and Refinement (SynthesizRR), which uses retrieval augmentation to introduce variety into the dataset synthesis process: as retrieved passages vary, the LLM is "seeded" with different content to generate its examples. We empirically study the synthesis of six datasets, covering topic classification, sentiment analysis, tone detection, and humor, requiring complex synthesis strategies. We find that SynthesizRR greatly improves lexical and semantic diversity, similarity to human-written text, and distillation performance, when compared to 32-shot prompting and four prior approaches. - BibTeX: ``` @inproceedings{divekar-durrett-2024-synthesizrr, title = {SynthesizRR: Generating Diverse Datasets with Retrieval Augmentation}, author = {Divekar, Abhishek and Durrett, Greg}, editor = {Al-Onaizan, Yaser and Bansal, Mohit and Chen, Yun-Nung}, booktitle = {Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing}, month = nov, year = {2024}, address = {Miami, Florida, USA}, publisher = {Association for Computational Linguistics}, url = {https://aclanthology.org/2024.emnlp-main.1071/}, doi = {10.18653/v1/2024.emnlp-main.1071}, pages = {19200-19227}, abstract = {It is often desirable to distill the capabilities of large language models (LLMs) into smaller student models due to compute and memory constraints. One way to do this for classification tasks is via dataset synthesis, which can be accomplished by generating examples of each label from the LLM. Prior approaches to synthesis use few-shot prompting, which relies on the LLM's parametric knowledge to generate usable examples. However, this leads to issues of repetition, bias towards popular entities, and stylistic differences from human text. In this work, we propose Synthesize by Retrieval and Refinement (SynthesizRR), which uses retrieval augmentation to introduce variety into the dataset synthesis process: as retrieved passages vary, the LLM is seeded with different content to generate its examples. We empirically study the synthesis of six datasets, covering topic classification, sentiment analysis, tone detection, and humor, requiring complex synthesis strategies. We find SynthesizRR greatly improves lexical and semantic diversity, similarity to human-written text, and distillation performance, when compared to 32-shot prompting and four prior approaches.} } ``` ### CorrSynth: A Correlated Sampling Method for Diverse Dataset Generation from LLMs - Venue: EMNLP 2024 (Main), Poster, 20.8% Accept Rate - Authors: Suhas S Kowshik*, Abhishek Divekar*, and Vijit Malik - Paper: https://arxiv.org/abs/2411.08553 - TL;DR: Correlated sampling across generations produces more diverse synthetic datasets from LLMs. - Abstract: Large language models (LLMs) have demonstrated remarkable performance in diverse tasks using zero-shot and few-shot prompting. Even though their capabilities of data synthesis have been studied well in recent years, the generated data suffers from a lack of diversity, less adherence to the prompt, and potential biases that creep into the data from the generator model. In this work, we tackle the challenge of generating datasets with high diversity, upon which a student model is trained for downstream tasks. Taking the route of decoding-time guidance-based approaches, we propose CorrSynth, which generates data that is more diverse and faithful to the input prompt using a correlated sampling strategy. Further, our method overcomes the complexity drawbacks of some other guidance-based techniques like classifier-based guidance. With extensive experiments, we show the effectiveness of our approach and substantiate our claims. In particular, we perform intrinsic evaluation to show the improvements in diversity. Our experiments show that CorrSynth improves both student metrics and intrinsic metrics upon competitive baselines across four datasets, showing the innate advantage of our method. - BibTeX: ``` @inproceedings{kowshik-etal-2024-corrsynth, title = {CorrSynth - A Correlated Sampling Method for Diverse Dataset Generation from LLMs}, author = {Kowshik, Suhas S and Divekar, Abhishek and Malik, Vijit}, editor = {Al-Onaizan, Yaser and Bansal, Mohit and Chen, Yun-Nung}, booktitle = {Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing}, month = nov, year = {2024}, address = {Miami, Florida, USA}, publisher = {Association for Computational Linguistics}, url = {https://aclanthology.org/2024.emnlp-main.899/}, doi = {10.18653/v1/2024.emnlp-main.899}, pages = {16076-16095}, abstract = {Large language models (LLMs) have demonstrated remarkable performance in diverse tasks using zero-shot and few-shot prompting. Even though their capabilities of data synthesis have been studied well in recent years, the generated data suffers from a lack of diversity, less adherence to the prompt, and potential biases that creep into the data from the generator model. In this work, we tackle the challenge of generating datasets with high diversity, upon which a student model is trained for downstream tasks. Taking the route of decoding-time guidance-based approaches, we propose CorrSynth, which generates data that is more diverse and faithful to the input prompt using a correlated sampling strategy. Further, our method overcomes the complexity drawbacks of some other guidance-based techniques like classifier-based guidance. With extensive experiments, we show the effectiveness of our approach and substantiate our claims. In particular, we perform intrinsic evaluation to show the improvements in diversity. Our experiments show that CorrSynth improves both student metrics and intrinsic metrics upon competitive baselines across four datasets, showing the innate advantage of our method.} } ``` ### Benchmarking datasets for Anomaly-based Network Intrusion Detection: KDD CUP 99 alternatives - Venue: IEEE ICCCS 2018, Oral - Authors: Abhishek Divekar, Meet Parekh, Vaibhav Savla, Rudra Mishra, and Mahesh Shirole - Paper: https://arxiv.org/abs/1811.05372 - TL;DR: Shows that classifiers trained on UNSW-NB15 match or beat those trained on the skewed KDD-99 and NSL-KDD benchmarks, making it a modern substitute. - Abstract: Machine Learning has been steadily gaining traction for its use in Anomaly-based Network Intrusion Detection Systems (A-NIDS). Research into this domain is frequently performed using the KDD~CUP~99 dataset as a benchmark. Several studies question its usability while constructing a contemporary NIDS, due to the skewed response distribution, non-stationarity, and failure to incorporate modern attacks. In this paper, we compare the performance for KDD-99 alternatives when trained using classification models commonly found in literature: Neural Network, Support Vector Machine, Decision Tree, Random Forest, Naive Bayes and K-Means. Applying the SMOTE oversampling technique and random undersampling, we create a balanced version of NSL-KDD and prove that skewed target classes in KDD-99 and NSL-KDD hamper the efficacy of classifiers on minority classes (U2R and R2L), leading to possible security risks. We explore UNSW-NB15, a modern substitute to KDD-99 with greater uniformity of pattern distribution. We benchmark this dataset before and after SMOTE oversampling to observe the effect on minority performance. Our results indicate that classifiers trained on UNSW-NB15 match or better the Weighted F1-Score of those trained on NSL-KDD and KDD-99 in the binary case, thus advocating UNSW-NB15 as a modern substitute to these datasets. - BibTeX: ``` @inproceedings{8586840, author={Divekar, Abhishek and Parekh, Meet and Savla, Vaibhav and Mishra, Rudra and Shirole, Mahesh}, booktitle={2018 IEEE 3rd International Conference on Computing, Communication and Security (ICCCS)}, title={Benchmarking datasets for Anomaly-based Network Intrusion Detection: KDD CUP 99 alternatives}, year={2018}, pages={1-8}, abstract={Machine Learning has been steadily gaining traction for its use in Anomaly-based Network Intrusion Detection Systems (A-NIDS). Research into this domain is frequently performed using the KDD CUP 99 dataset as a benchmark. Several studies question its usability while constructing a contemporary NIDS, due to the skewed response distribution, non-stationarity, and failure to incorporate modern attacks. In this paper, we compare the performance for KDD-99 alternatives when trained using classification models commonly found in literature: Neural Network, Support Vector Machine, Decision Tree, Random Forest, Naive Bayes and K-Means. Applying the SMOTE oversampling technique and random undersampling, we create a balanced version of NSL-KDD and prove that skewed target classes in KDD-99 and NSL-KDD hamper the efficacy of classifiers on minority classes (U2R and R2L), leading to possible security risks. We explore UNSW-NB15, a modern substitute to KDD-99 with greater uniformity of pattern distribution. We benchmark this dataset before and after SMOTE oversampling to observe the effect on minority performance. Our results indicate that classifiers trained on UNSW-NB15 match or better the Weighted F1-Score of those trained on NSL-KDD and KDD-99 in the binary case, thus advocating UNSW-NB15 as a modern substitute to these datasets.}, keywords={Training;Machine learning;Security;Benchmark testing;Probes;Feature extraction;Conferences;KDD-99;NSL-KDD;Network Intrusion Detection;Benchmarking;SMOTE;UNSW-NB15}, doi={10.1109/CCCS.2018.8586840}, month={Oct}, } ``` ### Amazon Machine Learning Conference AMLC is Amazon's annual internal machine learning conference, with roughly 1,000 submissions a year ([announcement](https://www.amazon.science/videos-webinars/amazons-annual-machine-learning-conference-featured-presentations-from-thought-leaders-within-academia)). Full texts are internal. - TrojanShot: One Learned Example Can Red-Team and Guard Large Language Models. Abhishek Divekar* and Vijit Malik*. AMLC 2025, Oral, 10% Accept Rate. TL;DR: A single learned in-context example can both red-team and guardrail large language models. Abstract: LLM-based applications increasingly rely on models served through third-party providers, where the model's own alignment is often the sole defense against misuse. In this regime, developers must implement both red-teaming and guardrails themselves, yet existing in-context methods address these objectives in isolation and do not refine demonstrations against the target LLM's responses. We propose TrojanShot, a unified framework that post-trains a demonstration generator via reinforcement learning to produce potent in-context examples for both attack (Trojan shots) and defense (Guardian shots), operating entirely at the prompt layer of the target LLM. On AdvBench, a single Trojan shot reaches 80.6% ASR on Mistral-7B, and averaged across four datasets TrojanShot exceeds 256-shot Many-Shot Jailbreaking by 14.6 p.p. while preserving human-readable prompts; ablations show RL post-training is the dominant factor in demonstration quality, with the untrained generator already surpassing 1-shot ICA. We demonstrate strong transferability, with a trojan-attacker trained on Mistral-7B reaching 81.2% ASR on Llama-3-70B, exceeding the 69.9% of an attacker trained on Llama-3-8B. Guardian shots reduce TrojanShot's own ASR by 48.9 p.p. with helpfulness within 1.1 p.p. of the undefended model. Our results hold for models whose intrinsic alignment is the primary defense. - BibTeX: ``` @inproceedings{10.1145/3799682.3841000, author = {Divekar, Abhishek and Malik, Vijit}, title = {TrojanShot: Reinforced Demonstrations Yield Sample-Efficient Jailbreaks and Guardrails for LLMs}, year = {2026}, isbn = {9798400725395}, publisher = {Association for Computing Machinery}, address = {New York, NY, USA}, url = {https://doi.org/10.1145/3799682.3841000}, doi = {10.1145/3799682.3841000}, abstract = {LLM-based applications increasingly rely on models served through third-party providers, where the model's own alignment is often the sole defense against misuse. In this regime, developers must implement both red-teaming and guardrails themselves, yet existing in-context methods address these objectives in isolation and do not refine demonstrations against the target LLM's responses. We propose TrojanShot, a unified framework that post-trains a demonstration generator via reinforcement learning to produce potent in-context examples for both attack (Trojan shots) and defense (Guardian shots), operating entirely at the prompt layer of the target LLM. On AdvBench, a single Trojan shot reaches 80.6% ASR on Mistral-7B, and averaged across four datasets TrojanShot exceeds 256-shot Many-Shot Jailbreaking by 14.6 p.p. while preserving human-readable prompts; ablations show RL post-training is the dominant factor in demonstration quality, with the untrained generator already surpassing 1-shot ICA. We demonstrate strong transferability, with a trojan-attacker trained on Mistral-7B reaching 81.2% ASR on Llama-3-70B, exceeding the 69.9% of an attacker trained on Llama-3-8B. Guardian shots reduce TrojanShot's own ASR by 48.9 p.p. with helpfulness within 1.1 p.p. of the undefended model. Our results hold for models whose intrinsic alignment is the primary defense. Warning: this paper contains harmful model responses for academic reference.}, booktitle = {Proceedings of the 35th ACM International Conference on Information and Knowledge Management}, pages = {1012-1023}, numpages = {12}, keywords = {large language models, reinforcement learning, in-context learning, llm safety, red-teaming, defense, trojanshot, guardianshot}, location = {Rome, Italy}, series = {CIKM '26} } ``` - SynthesizRR: Generating Diverse Datasets with Retrieval Augmentation (AMLC version). Abhishek Divekar and Greg Durrett. AMLC 2024, Oral, 10% Accept Rate. TL;DR: Retrieval augmentation makes LLM-generated training data more diverse and closer to human-written text. Abstract: 'It is often desirable to distill the capabilities of large language models (LLMs) into smaller student models due to compute and memory constraints. One way to do this for classification tasks is via dataset synthesis, which can be accomplished by generating examples of each label from the LLM. Prior approaches to synthesis use few-shot prompting, which relies on the LLM''s parametric knowledge to generate usable examples. However, this leads to issues of repetition, bias towards popular entities, and stylistic differences from human text. In this work, we propose Synthesize by Retrieval and Refinement (SynthesizRR), which uses retrieval augmentation to introduce variety into the dataset synthesis process: as retrieved passages vary, the LLM is "seeded" with different content to generate its examples. We empirically study the synthesis of six datasets, covering topic classification, sentiment analysis, tone detection, and humor, requiring complex synthesis strategies. We find that SynthesizRR greatly improves lexical and semantic diversity, similarity to human-written text, and distillation performance, when compared to 32-shot prompting and four prior approaches.' - BibTeX: ``` @inproceedings{divekar-durrett-2024-synthesizrr, title = {SynthesizRR: Generating Diverse Datasets with Retrieval Augmentation}, author = {Divekar, Abhishek and Durrett, Greg}, editor = {Al-Onaizan, Yaser and Bansal, Mohit and Chen, Yun-Nung}, booktitle = {Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing}, month = nov, year = {2024}, address = {Miami, Florida, USA}, publisher = {Association for Computational Linguistics}, url = {https://aclanthology.org/2024.emnlp-main.1071/}, doi = {10.18653/v1/2024.emnlp-main.1071}, pages = {19200-19227}, abstract = {It is often desirable to distill the capabilities of large language models (LLMs) into smaller student models due to compute and memory constraints. One way to do this for classification tasks is via dataset synthesis, which can be accomplished by generating examples of each label from the LLM. Prior approaches to synthesis use few-shot prompting, which relies on the LLM's parametric knowledge to generate usable examples. However, this leads to issues of repetition, bias towards popular entities, and stylistic differences from human text. In this work, we propose Synthesize by Retrieval and Refinement (SynthesizRR), which uses retrieval augmentation to introduce variety into the dataset synthesis process: as retrieved passages vary, the LLM is seeded with different content to generate its examples. We empirically study the synthesis of six datasets, covering topic classification, sentiment analysis, tone detection, and humor, requiring complex synthesis strategies. We find SynthesizRR greatly improves lexical and semantic diversity, similarity to human-written text, and distillation performance, when compared to 32-shot prompting and four prior approaches.} } ``` - Squeezing the last DRiP: AutoML for cost-constrained Product classification. Abhishek Divekar*, Gaurav Manchanda*, Prit Raj, Abhishek Das, Karan Tanwar, Akshay Jagatap, Vinayak Puranik, Jagannathan Srinivasan, Ramakrishna Nalam, and Nikhil Rasiwasia. AMLC 2021, Poster, 30% Accept Rate. TL;DR: AutoML that finds high-performing models within an explicit production cost budget, with a cost-performance tradeoff curve. Abstract: Recent progress in automated machine learning (AutoML) has shown that both hyperparameter search and stacking models can achieve performance that beats the median Kaggle competitor on standard classification metrics. However, solutions that directly optimize the cost of executing such models with minimal loss of performance have been sparingly explored. Current techniques lack a priori knowledge of the deployment environment and its constraints, such as request batch-size or prediction latency. This affects a critical subset of users: business teams with cost-sensitive use-cases, who cannot meet their goals if an AutoML-suggested model is expensive during training or inference. To bridge this gap, we propose DRiP, a versatile ML framework that encapsulates the phases needed for an automated system to select and iteratively refine candidate models while being constrained by an inference budget. Existing tools have individual capabilities that map to phases in our framework, but lack the overall capability to optimize against such constraints. We present our implementation of this framework as a new AutoML tool. Across 38 product classification datasets and various business use-cases, DRiP obtains 99.96% of the ROC-AUC performance of the best SOTA AutoML systems (AutoGluon and H2O.ai) at 37% of the cost. When tuned to minimize cost, DRiP offers better cost and performance than comparably optimized AutoML systems (average 0.21 ROC-AUC increase at 44% of the cost of distilled AutoGluon). If no cost constraints are imposed, Unrestricted DRiP provides the best overall performance (98.42 ROC-AUC vs 98.18 for the next-best AutoML system). Although we evaluate classification, the framework can optimize any machine learning task with well-defined performance and cost. - CPP Multimodal AutoML Corpus and Benchmark. Andrew Borthwick, Abhishek Divekar, Nick Erickson, Fayaz Ahmed Farooque, Oleg Kim, Nikhil Rasiwasia, and Ethan Xu. AMLC 2021 (Workshop), Oral. TL;DR: A corpus of 40 multimodal binary classification datasets from real Amazon product problems, for benchmarking AutoML. Abstract: A collection of 40 binary classification datasets was acquired from an integrated AutoML, active learning, and human labeling system for Amazon products known as ”CPP AutoML”. Each dataset consists of an identical schema of 39 attributes including numeric, categorical, text, image and date attributes. Each dataset represents a real business problem that is being solved by the CPP AutoML platform. In this paper, we discuss the construction and structure of this corpus. We also discuss the challenges of evaluating AutoML algorithms for the “hands off the wheel” business requirements of CPP AutoML. Finally, we present short descriptions and benchmark metrics over this corpus for a collection of algorithms. These algorithms include the CPP AutoML production baseline, two common machine learning baselines, two publicly available Amazon AutoML solutions (AutoGluon and SageMaker AutoPilot), and two novel AutoML solutions. The novel AutoML solutions exhibit particularly strong performance relative to the production baseline. One of these solutions exercises new functionality added to AutoGluon to stack and ensemble a ResNet image model fine tuned on raw image features, used in addition to the tabular models already trained in standard AutoGluon. The other solution, EPS-Ensemble, combines standard gradient-boosted trees and logistic regression models with Transformer networks pre-trained on the Amazon catalog using different self-supervised objectives. - Entity Prediction Service: a configurable, end-to-end AutoML system. Gaurav Manchanda*, Abhishek Divekar*, Akshay Jagatap, Prit Raj, Vinayak Puranik, Nikhil Rasiwasia, Ramakrishna Nalam, and Jagannathan Srinivasa. AMLC 2020 (Workshop), Poster. TL;DR: A configurable, end-to-end AutoML system that lets Amazon business teams classify products into taxonomies without ML expertise. Abstract: Business teams at Amazon often need to classify products into different taxonomies such as GL, item type keyword (ITK), category/sub-category, browse node, tax code, export-compliance-code and hazmat. Due to the lack of ML expertise, these teams end up relying on human auditors or manually codify rules, which is not scalable or do not work in cases having data with high diversity. Existing ML solutions for AutoML product classification are either stand-alone applications that push the burden of model productionization onto users (e.g. SageMaker Autopilot and AutoGluon), or are production-friendly, but lack state-of-the-art AutoML capabilities and do not leverage the agility offered by modern tech ecosystems. In this paper, we present Entity Prediction Service (EPS), a configurable product classification solution designed to serve the end-to-end needs of Amazon teams. Leveraging the robust ecosystem of AWS services and Docker, EPS automatically fetches and pre-processes data from internal data sources, trains and tunes models, performs inference, and enables one-click deployment into production. Each step offers a granular level of configurability, with default parameters backed by a robust set of scientific benchmarks. This helps serve customers across the spectrum of Machine Learning expertise, enabling Business Associates, SDEs and Applied Scientists to build high quality product classification models and deploy them on Amazon systems for continuous classification. ## Product launches - KeywordGym for Amazon advertising (2025 to now). My role: Led the science: designed the training environment, reward and guardrails, and launched the keyword models in a live campaign experiment. A reinforcement-learning environment that post-trains language models to generate advertising keyword sets for an advertiser's primary objective (such as orders or clicks), with a group-relative guardrail that protects secondary metrics such as return on ad spend. - PRECISE in GlowRoad search (Oct 2024). My role: Built the statistical method and deployed it to evaluate GlowRoad search; first author of the AAAI 2026 paper. Measures search relevance from about 100 human-labelled queries plus LLM judgments, correcting for the LLM judge's bias. - Amazon Rufus, India launch (2024). My role: Led red-teaming for the India launch, testing Rufus against India-specific compliance policies. Amazon's LLM-powered shopping assistant, used by 100M+ customers. - AutoML platform for product classification (2019 to 2023). My role: Lead science contributor: designed and built its data-processing nodes (160K lines of Python). Internal AutoML platform that deployed 1,000+ models across 14 Amazon websites and updated 1B+ product attributes. ## AI Olympiad The [International Olympiad in Artificial Intelligence (IOAI) [[ARROW]]](https://ioai-official.org/) is an annual science olympiad for high-school students. Its third edition, IOAI 2026 in Astana, Kazakhstan, had 108 countries and territories represented, and was the first to include an AI Models Track. ### Chair of IOAI²: AI Models Track; member of IOAI 2026 Scientific Committee IOAI 2026 · Astana, Kazakhstan As an ISC member, I helped propose and curate the IOAI 2026 tasks and served on the IOAI Jury in Astana, which proctored the competition. I also co-created and chaired the inaugural IOAI²: AI Models Track, and led its organizing committee. This first-of-its-kind competition measured the ability of agentic AI systems to solve our Olympiad-level ML modeling tasks, fully autonomously and with no human intervention. As chair, I spearheaded recruitment of 14 AI research organizations participating in the track across 3 continents, including UCLA, Peking University, Sakana AI, MBZUAI IFM, Shanghai AI Laboratory, Darwin Robotics and several AI neolabs. I led development of rules and proctoring of the track, announced medals, and co-wrote the final report with co-founder and Chair of the IOAI Board, Elena Marinova. The founding AI participants of the IOAI² AI Models Track: the logos of the 14 labs and teams whose autonomous AI systems competed in the inaugural AI Models Track. - IOAI 2026 Team: https://ioai-official.org/republic-of-kazakhstan/international-jury/ - IOAI² Rules: https://ioai-official.org/ai-models-track/ - IOAI² Results: https://ioai-official.org/ai-models-track/ioai%C2%B2-2026-report/ - IOAI² GitHub: https://github.com/IOAI-official/IOAI-AI-Models-Track ### Team Lead, Team India IOAI 2025 · Beijing, China I served as Team Lead (Olympiad Coach) for Team India at its debut participation at the International Olympiad in AI (IOAI). I helped select, mentor and coach the AI olympians, ran timed six-hour mock contests, and represented India at the IOAI General Assembly. I also designed Team India's peacock logo (right; each feather represents one olympian who participated in 2025). The team of 8 won 6 medals, placing 3rd of 63 countries. Golds: Roumak Das (rank 11), Arjun Tyagi (18), Soham Sen (19). Silvers: Rayan Banerjee (37), Samik Goyal (41). Bronze: Himaneesh Sompalle (77). Honorable Mention: Ishaan Gangwani (152). Press coverage: [The Hindu BusinessLine](https://www.thehindubusinessline.com/news/indian-team-bags-third-spot-in-its-debut-appearance-at-ai-olympiad/article69960200.ece), [The Times of India](https://timesofindia.indiatimes.com/city/chennai/indian-students-win-3-gold-medals-at-intl-ai-olympiad/articleshow/123474957.cms), [Navbharat Times](https://navbharattimes.indiatimes.com/tech/ai-news/india-secured-third-postion-in-international-olympiad-in-artificial-intelligence-ahead-of-china-and-america/articleshow/123436355.cms), [Zee News](https://zeenews.india.com/hindi/technology/india-shines-at-international-ai-olympiad-2025-beats-us-and-china-to-win-3rd-place/2891483), [India.com](https://www.india.com/hindi-news/gallery-hindi/india-occupy-3rd-position-in-international-ai-olympiad-beat-us-china-8029719/india-vs-usa-china-ai-olympiad/) Team India at IOAI 2025 in Beijing: the eight olympians with their medals, with Abhishek Divekar on the right. - IOAI Results: https://ioai-official.org/china-2025/results-2025/ - Press coverage: https://timesofindia.indiatimes.com/city/chennai/indian-students-win-3-gold-medals-at-intl-ai-olympiad/articleshow/123474957.cms - INAIO Team: https://www.inaio.in/our-team ### Technical Committee, ACM India AI Olympiad (INAIO 2025) INAIO 2025 · Hyderabad, India I helped select India's national team: I curated the Stage-2 problems attempted by 200+ students, and coached and judged the 22 Stage-4 finalists at IIIT Hyderabad. - INAIO Team: https://www.inaio.in/our-team ## Open source I led the open-sourcing of FMCore, Bears and Concurry on Amazon Science. - IOAI² AI Models Track: Task adaptations, scoring, competition rules and Jury precedents, for reuse by other olympiads. [GitHub](https://github.com/IOAI-official/IOAI-AI-Models-Track) - Concurry: Scaling AI research and production workloads in Python. [Docs](https://amazon-science.github.io/concurry/), [GitHub](https://github.com/adivekar-utexas/concurry) - FMCore: Running GenAI models at every scale and modality. [GitHub](https://github.com/adivekar-utexas/fmcore) - Bears: Dataframe sharding, streaming and processing for ML. [GitHub](https://github.com/adivekar-utexas/bears) - AI Learning Gems: Textbook-style AI notes, researched by LLMs and curated by me. [GitHub](https://github.com/AI-Learning-Gems) ## Awards - Jan 2026: IAAI 2026 Deployed Application Award, AAAI 2026, Innovative Applications of AI (IAAI) track, Singapore. For PRECISE: Reducing the Bias of LLM Evaluations using Prediction-Powered Ranking Estimation (with Anirban Majumder). [See the paper](https://adivekar.com/#precise-aaai-2026) - 2025: Top 7 GenAI Scientists in India, Analytics Vidhya, DataHack Summit 2025 [Read the announcement](https://www.analyticsvidhya.com/blog/2025/08/dhs-day-2/) - 2025: Top 7 GenAI Community Contributors, Analytics Vidhya, DataHack Summit 2025 - 2025: Team Lead, Team India at IOAI 2025, 3 Gold, 2 Silver, 1 Bronze, 1 Honorable Mention; 3rd of 63 by medal table [See the official results](https://ioai-official.org/china-2025/results-2025/) - Aug 2017: Winner, Amazon Machine Learning Challenge, Chennai, out of 300 participants Photos (Top 7 GenAI Scientists in India, Analytics Vidhya DataHack Summit 2025.): - Receiving the award from Manish Gupta, Director, Google DeepMind. (Abhishek Divekar receiving the Top 7 GenAI Scientists in India award from Manish Gupta, Director at Google DeepMind, at Analytics Vidhya DataHack Summit 2025) - With Pratyush Kumar, Co-founder, Sarvam.ai. (Abhishek Divekar with Pratyush Kumar, Co-founder of Sarvam.ai, at the Top 7 GenAI Scientists in India award, Analytics Vidhya DataHack Summit 2025) ## Press & Media - Aug 2025: Team India places 3rd at its debut International Olympiad in AI. Coverage of the team I led at IOAI 2025, Beijing (3 Gold, 2 Silver, 1 Bronze, 1 Honorable Mention). Read the coverage in [The Hindu BusinessLine](https://www.thehindubusinessline.com/news/indian-team-bags-third-spot-in-its-debut-appearance-at-ai-olympiad/article69960200.ece), [The Times of India](https://timesofindia.indiatimes.com/city/chennai/indian-students-win-3-gold-medals-at-intl-ai-olympiad/articleshow/123474957.cms), [Navbharat Times](https://navbharattimes.indiatimes.com/tech/ai-news/india-secured-third-postion-in-international-olympiad-in-artificial-intelligence-ahead-of-china-and-america/articleshow/123436355.cms), [Zee News](https://zeenews.india.com/hindi/technology/india-shines-at-international-ai-olympiad-2025-beats-us-and-china-to-win-3rd-place/2891483) and [India.com](https://www.india.com/hindi-news/gallery-hindi/india-occupy-3rd-position-in-international-ai-olympiad-beat-us-china-8029719/india-vs-usa-china-ai-olympiad/). - Aug 2025: DataHack Summit 2025, Day 2: talk on synthetic data and Top 7 GenAI Scientists award. Analytics Vidhya's Day 2 recap covers my talk, The Promise and Pitfalls of Synthetic Data Generation, and lists me among the AV Luminary Award winners. Read in [Analytics Vidhya](https://www.analyticsvidhya.com/blog/2025/08/dhs-day-2/). - 2022: Amazon India announces second edition of Machine Learning Summer School. Covers the summer school where I taught. Read in [The Times of India](https://timesofindia.indiatimes.com/gadgets-news/amazon-india-announces-second-edition-of-machine-learning-summer-school-how-to-apply-and-other-details/articleshow/92076056.cms). - 2022: Being a machine learning scientist at Amazon India. Video interview about my work at Amazon India. [Watch the video interview](https://youtu.be/15sa6OeIWJg) ## Service - 2025: Area Chair, Amazon Machine Learning Conference (AMLC) - 2025: Reviewer, ACL - 2025: Reviewer, EMNLP - 2025: Reviewer, ICLR - 2025: Reviewer, AAAI - 2025: Reviewer, COLM - 2025: Reviewer, WSDM - 2025: Reviewer, SIGIR - 2025: Reviewer, WWW - 2025: Proposal reviewer, Amazon Research Awards, AWS Agentic AI call for proposals [See the call for proposals](https://www.amazon.science/research-awards/call-for-proposals/aws-agentic-ai-call-for-proposals-fall-2025) - 2022: Reviewer, AMLC, NLP and ML Tools tracks - 2022: Reviewer, AIMLSys - 2022: Instructor, Amazon ML Summer School, about 3,000 Indian undergraduates. Taught the supervised-learning module: decision trees, bagging and boosting. - Ongoing: Hiring and mentoring, Amazon. 200+ interviews for Applied Scientist roles. I mentor junior scientists and interns. ## Writing Notes on research, careers and engineering. - [Setting Up JupyterHub for Your AI Research Team on a New Linux Machine](https://adivekar.com/writing/jupyterhub-for-ai-research-teams/) (25 Oct 2025, 18 min; tags: engineering, infrastructure, guide): A step-by-step guide to a shared JupyterHub: per-user home directories, shared read-only conda environments, personal envs, Jupyter AI, and Ray. - [Tips for a Successful Resume and Referral at Amazon](https://adivekar.com/writing/tips-for-amazon-referrals/) (09 Sep 2024, 9 min; tags: career, hiring): My rules for referrals, and how to write a resume that survives a hiring manager's first read: role fit, impact datapoints, ordering, and one page. - [Doing an MS Thesis While Working Full-Time as an ML Scientist](https://adivekar.com/writing/thesis-while-working-full-time/) (15 Dec 2023, 6 min; tags: research, career, education): What it was really like to do an NLP thesis at UT Austin during 2023 while working full-time at Amazon, and why I would do it again. - [Amazon Engineer to ML Scientist: Notes on a 2-Year Journey of Growth](https://adivekar.com/writing/engineer-to-ml-scientist/) (09 Dec 2021, 14 min; tags: career, machine-learning, research): How I moved from software engineering to Applied Science at Amazon without a Masters or PhD: personal investment, ML skills, knowing the job, and learning to be a scientist. ## Speaking Talks, oral presentations, lectures and interviews, with slides, photos and video. Newest first. - [PRECISE: Reducing the Bias of LLM Evaluations using Prediction-Powered Ranking Estimation](https://adivekar.com/speaking/#precise-aaai-2026-oral). Oral presentation, 24 January 2026, AAAI Conference on Artificial Intelligence (AAAI 2026), Innovative Applications of AI track, Singapore. Topics: LLM evaluation, LLM-as-a-judge, oral. Oral presentation in the Innovative Applications of AI track (19.2% acceptance rate). PRECISE combines a small set of human relevance labels with LLM judgments, using prediction-powered inference to estimate ranking metrics such as Precision@K while correcting for LLM-judge bias. Paper: https://arxiv.org/abs/2601.18777 - [The Promise and Pitfalls of Synthetic Data Generation](https://adivekar.com/speaking/#dhs-2025-synthetic-data). Invited talk and hands-on session, 21 August 2025, DataHack Summit 2025, Analytics Vidhya. Topics: synthetic data, LLMs, invited talk. Despite rapid advances in generative AI, representative high-quality data remains a major bottleneck. This talk reviewed how LLM developers now use synthetic data, where synthetic data helps and where it fails, and how to diagnose the quality and diversity of generated datasets, drawing on SynthesizRR and CorrSynth (EMNLP 2024). A companion hands-on session covered synthetic-data diagnostics in code. Photos: 6 on the Speaking page. Session page (Analytics Vidhya): https://www.analyticsvidhya.com/datahacksummit-2025/sessions/the-promise-and-pitfalls-of-synthetic-data-generationno-title - [Flexible AutoML: Accelerating AutoML adoption across Amazon](https://adivekar.com/speaking/#litmus-aimlsys-2022-talk). Talk, 07 October 2022, 2nd International Conference on AI-ML Systems (AIMLSys 2022), Virtual. Topics: AutoML, ML platforms. Current AutoML systems consider only two scenarios: (i) AutoML model meets performance bar, or (ii) model misses performance bar. Efforts have been dedicated to improving this ratio, but common issues are not addressed: (iii) model missed performance bar by 1% (iv) model is too slow in production (v) Custom model needs 6+ months to deploy. Different AutoML user-personas (Data Scientists, Engineers, Non-Tech) face different issues depending on their background. Flexible AutoML is a paradigm which addresses the needs of all personas. We present an experimentation platform, Litmus, which provides convenient interfaces for experimentation for each user-persona, is extensible to new ML paradigms, and scales to large models and datasets. We further discuss how Litmus accelerates AutoML adoption across Amazon. Slides: https://adivekar.com/files/AIMLSys_2022_demo_vF.pdf - [Supervised Learning (Decision Trees, Bagging and Boosting algorithms)](https://adivekar.com/speaking/#ml-summer-school-2022-lecture). Lecture, 14 June 2022, Amazon ML Summer School 2022, Virtual. Topics: teaching, supervised learning. Amazon ML Summer School 2022 was an initiative which enrolled ~3,000 Indian undergraduate students and helped them learn key ML technologies from Amazon Scientists. I taught a module on Supervised Learning covering Decision Trees, Bagging and Boosting algorithms, followed by a 3-hour Q&A session to clarify student questions. Slides: https://adivekar.com/files/ML-Summer-School-2022-Supervised-Learning-part-2-Abhishek-slides.pdf - [Being a Machine Learning Scientist at Amazon India](https://adivekar.com/speaking/#amazon-india-scientist-interview). Interview, 2022, Amazon. Topics: careers, video. Video interview about my work as a machine learning scientist at Amazon India. Video: https://youtu.be/15sa6OeIWJg - [Squeezing the last DRiP: AutoML for Cost-constrained Product Classification](https://adivekar.com/speaking/#drip-ard-2021-talk). Talk, 26 October 2021, Amazon Research Days 2021 conference, Virtual. Topics: AutoML, production ML. Current AutoML research aims to minimize the Discovery time of high-performing models, e.g. "find best model within 30 mins". However, ideally most models are trained, and used in production for months before refresh, meaning the operational cost of running an AutoML model in production far exceeds the one-time discovery cost. Instead, can AutoML systems discover high-performing models which operate within an explicit budget? We propose a new AutoML paradigm, DRiP (Discover-Refine-Productionize) which not only allows cost-backwards optimization, but produces a cost-performance tradeoff curve for users to choose an appropriate point. We compare to AutoGluon v0.2 and find that DRiP AutoML can be tuned to achieve: (i) On-par performance at low cost (ii) Minimum overall cost (iii) Maximum overall performance. Slides: https://adivekar.com/files/Squeezing_the_last_DRiP_ARD_2021_slides.pdf