Mert Yuksekgonul

I'm a second-year PhD student in Computer Science at Stanford University. I am lucky to be advised by James Zou and Carlos Guestrin.

These days I am interested in enabling safer use and a greater understanding of deep learning. I am broadly interested in explaining model behavior through human understandable terms, designing ways to intervene on models, and improving multimodal understanding. I really enjoy thinking about abstraction, concepts, and compositionality in humans and machines.

I was extremely fortunate to have many wonderful mentors in my undergraduate years. I worked with Prof. Mackenzie Mathis, Prof. Alexander Mathis and Prof. Matthias Bethge as part of Bethge Lab(Tübingen/MPI) and Adaptive Motor Control Lab (Harvard/EPFL). My last undergrad research gig was with Dr. Xavier Boix as part of Poggio Lab(MIT) and Sinha Lab(MIT). I started Machine Learning research with Prof. Mustafa Baydogan at Bogazici University.

[ Email / Github / Twitter / Google Scholar ]

News

Jan 2023: 1 Oral and 1 Spotlight @ICLR23! When and why vision-language models behave like bags-of-words, and what to do about it? (Oral) and Post-hoc Concept Bottleneck Models (Spotlight) are accepted to ICLR 2023!
Nov 2022: We released Holistic Evaluation of Language Models (HELM) with 50+ collaborators at Stanford CRFM.
Oct 2022: I was named an outstanding reviewer for NeurIPS2022 Datasets and Benchmarks track. This was my first ever reviewing acknowledgement, feels nice! :)
Oct 2022: New preprint - When and why vision-language models behave like bags-of-words, and what to do about it? is on ArXiv now. I enjoyed thinking about compositionality and bag-of-wordness of vision-language models, check it out!
Sep 2022: Paper @ NeurIPS 2022 - Our work SkinCon: A skin disease dataset densely annotated by domain experts for fine-grained debugging and analysis was accepted to NeurIPS 2022 Datasets and Benchmarks Track.
May 2022: Paper @ ICML 2022 - Our work Meaningfully debugging model mistakes using conceptual counterfactual explanations was accepted to ICML 2022. This work happens to be the first conference paper from my PhD!
April 2022: Paper @ ICLR 2022, PAIR2Struct - Our work Post-hoc Concept Bottleneck Models was accepted to ICLR2022 Pair2Struct Workshop.
September 2021: Moved to Stanford, and started my PhD!
June 2021: Graduated from Bogazici University, with a BS in Computer Engineering and a BS in Industrial Engineering (Operations Research).

Selected Publications

	When and why vision-language models behave like bags-of-words, and what to do about it? Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, James Zou Oral @ ICLR 2023 (Top 5% of all accepted papers) [ Paper , Code ] ▶ Show Description Recent work [and many tweet threads] suggests that Vision-Language models(VLMs) such as CLIP do not fare well with compositional understanding. Here, we first propose a large-scale benchmark, ARO(Attribution, Relation and Order) to evaluate fine-grained relational, attributive, and order understanding. Why did the BoW-like behavior is not reflected in the retrieval evaluations (e.g. COCO/Flickr30k), where the datasets contain rich compositional structure? We propose interesting experiments to show that models do not need to do well with compositions to perform well on these tasks. Similarly, this can explain why contrastive-pretrained models may be exploiting this shortcut, and we should be careful about it. Following this intuition, we propose composition-aware negative mining. Check out our work !
	Post-hoc Concept Bottleneck Models Mert Yuksekgonul, Maggie Wang, James Zou Spotlight @ ICLR2023 (Top 25% of all accepted papers) [ Paper, Code ] ▶ Show Description Concept Bottleneck Models are very cool! But it's hard to find large training datasets with concept annotations, or match the performance of unrestricted neural nets with limited bottlenecks. In this work, we address practical limitations of CBMs by introducing Post-hoc Concept Bottleneck models (PCBMs). Further, we have a user study where humans can improve PCBMs via concept-level feedback.
	Beyond Confidence: Reliable Models Should Also Quantify Atypicality Mert Yuksekgonul, Linjun Zhang, James Zou, Carlos Guestrin Contributed Talk @ ICLR 2023 Workshop on Trustworthy Machine Learning [ Code soon.] ▶ Show Description While most machine learning models can provide confidence in their predictions, confidence is insufficient to understand and use the model's uncertainty reliably. In this work, we investigate the relationship between how atypical~(or rare) a sample is and the reliability of a model's confidence for this sample. Read the paper for interesting connections between atypicality and uncertainty!
	Leveraging medical Twitter to build a visual–language foundation model for pathology AI Zhi Huang, Federico Bianchi, Mert Yuksekgonul, Thomas Montine, James Zou Under Review at Nature Medicine [ Preprint, Demo ] ▶ Show Description We collect data from Twitter (yes, Twitter) + LAION and release the largest text-image pathology dataset: OpenPath. We also release PLIP, a CLIP-variant for pathology which gives exciting performance in 0-shot, transfer learning, and retrieval for Path.
	Meaningfully debugging model mistakes using conceptual counterfactual explanations Abubakar Abid, Mert Yuksekgonul, James Zou ICML 2022 [ Paper , Code ] ▶ Show Description We use human understandable concepts and counterfactual explanations (we call Conceptual Counterfactual Explanations) to debug model mistakes and reveal biases of a model.

News

Selected Publications

► More Publications