Aaron Kirtland

Aaron Kirtland

Washington University in St. Louis

H-index: 3

North America-United States

About Aaron Kirtland

Aaron Kirtland, With an exceptional h-index of 3 and a recent h-index of 3 (since 2020), a distinguished researcher at Washington University in St. Louis,

His recent articles reflect a diverse array of research interests and contributions to the field:

REBUS: A Robust Evaluation Benchmark of Understanding Symbols

An unstructured mesh approach to nonlinear noise reduction for coupled systems

Inverse Scaling: When Bigger Isn't Better

The polynomial learning with errors problem and the smearing condition

The Ring Learning With Errors Problem: Spectral Distortion

Aaron Kirtland Information

University

Washington University in St. Louis

Position

___

Citations(all)

59

Citations(since 2020)

57

Cited By

14

hIndex(all)

3

hIndex(since 2020)

3

i10Index(all)

2

i10Index(since 2020)

2

Email

University Profile Page

Washington University in St. Louis

Top articles of Aaron Kirtland

REBUS: A Robust Evaluation Benchmark of Understanding Symbols

Authors

Andrew Gritsevskiy,Arjun Panickssery,Aaron Kirtland,Derik Kauffman,Hans Gundlach,Irina Gritsevskaya,Joe Cavanagh,Jonathan Chiang,Lydia La Roux,Michelle Hung

Journal

arXiv preprint arXiv:2401.05604

Published Date

2024/1/11

We propose a new benchmark evaluating the performance of multimodal large language models on rebus puzzles. The dataset covers 333 original examples of image-based wordplay, cluing 13 categories such as movies, composers, major cities, and food. To achieve good performance on the benchmark of identifying the clued word or phrase, models must combine image recognition and string manipulation with hypothesis testing, multi-step reasoning, and an understanding of human cognition, making for a complex, multimodal evaluation of capabilities. We find that proprietary models such as GPT-4V and Gemini Pro significantly outperform all other tested models. However, even the best model has a final accuracy of just 24%, highlighting the need for substantial improvements in reasoning. Further, models rarely understand all parts of a puzzle, and are almost always incapable of retroactively explaining the correct answer. Our benchmark can therefore be used to identify major shortcomings in the knowledge and reasoning of multimodal large language models.

An unstructured mesh approach to nonlinear noise reduction for coupled systems

Authors

Aaron Kirtland,Jonah Botvinick-Greenhouse,Marianne DeBrito,Megan Osborne,Casey Johnson,Robert S Martin,Samuel J Araki,Daniel Q Eckhardt

Journal

SIAM Journal on Applied Dynamical Systems

Published Date

2023/12/31

To address noise inherent in electronic data acquisition systems and real-world sources, Araki et al. [Phys. D, 417 (2021), 132819] demonstrated a grid-based nonlinear technique to remove noise from a chaotic signal, leveraging a clean high-fidelity signal from the same dynamical system and ensemble averaging in multidimensional phase space. This method achieved denoising of a time series data with 100% added noise but suffered in regions of low data density. To improve this grid-based method, here an unstructured mesh based on triangulations and Voronoi diagrams is used to accomplish the same task. The unstructured mesh more uniformly distributes data samples over mesh cells to improve the accuracy of the reconstructed signal. By empirically balancing bias and variance errors in selecting the number of unstructured cells as a function of the number of available samples, the method achieves …

Inverse Scaling: When Bigger Isn't Better

Authors

Ian R McKenzie,Alexander Lyzhov,Michael Pieler,Alicia Parrish,Aaron Mueller,Ameya Prabhu,Euan McLean,Aaron Kirtland,Alexis Ross,Alisa Liu,Andrew Gritsevskiy,Daniel Wurgaft,Derik Kauffman,Gabriel Recchia,Jiacheng Liu,Joe Cavanagh,Max Weiss,Sicong Huang,The Floating Droid,Tom Tseng,Tomasz Korbak,Xudong Shen,Yuhui Zhang,Zhengping Zhou,Najoung Kim,Samuel R Bowman,Ethan Perez

Journal

arXiv preprint arXiv:2306.09479

Published Date

2023/6/15

Work on scaling laws has found that large language models (LMs) show predictable improvements to overall loss with increased scale (model size, training data, and compute). Here, we present evidence for the claim that LMs may show inverse scaling, or worse task performance with increased scale, e.g., due to flaws in the training objective and data. We present empirical evidence of inverse scaling on 11 datasets collected by running a public contest, the Inverse Scaling Prize, with a substantial prize pool. Through analysis of the datasets, along with other examples found in the literature, we identify four potential causes of inverse scaling: (i) preference to repeat memorized sequences over following in-context instructions, (ii) imitation of undesirable patterns in the training data, (iii) tasks containing an easy distractor task which LMs could focus on, rather than the harder real task, and (iv) correct but misleading few-shot demonstrations of the task. We release the winning datasets at https://inversescaling.com/data to allow for further investigation of inverse scaling. Our tasks have helped drive the discovery of U-shaped and inverted-U scaling trends, where an initial trend reverses, suggesting that scaling trends are less reliable at predicting the behavior of larger-scale models than previously understood. Overall, our results suggest that there are tasks for which increased model scale alone may not lead to progress, and that more careful thought needs to go into the data and objectives for training language models.

The polynomial learning with errors problem and the smearing condition

Authors

Liljana Babinkostova,Ariana Chin,Aaron Kirtland,Vladyslav Nazarchuk,Esther Plotnick

Journal

Journal of Mathematical Cryptology

Published Date

2022/8/10

As quantum computing advances rapidly, guaranteeing the security of cryptographic protocols resistant to quantum attacks is paramount. Some leading candidate cryptosystems use the learning with errors (LWE) problem, attractive for its simplicity and hardness guaranteed by reductions from hard computational lattice problems. Its algebraic variants, ring-learning with errors (RLWE) and polynomial learning with errors (PLWE), gain efficiency over standard LWE, but their security remains to be thoroughly investigated. In this work, we consider the “smearing” condition, a condition for attacks on PLWE and RLWE introduced in Elias et al. We expand upon some questions about smearing posed by Elias et al. and show how smearing is related to the coupon collector’s problem. Furthermore, we develop an algorithm for computing probabilities related to smearing. Finally, we present a smearing-based algorithm …

The Ring Learning With Errors Problem: Spectral Distortion

Authors

L Babinkostova,Ariana Chin,Aaron Kirtland,Vladyslav Nazarchuk,Esther Plotnick

Journal

arXiv preprint arXiv:2007.13189

Published Date

2020/7/26

We answer a question posed by Y. Elias et al. in [8] about possible spectral distortions of algebraic numbers. We provide a closed form for the spectral distortion of certain classes of cyclotomic polynomials. Moreover, we present a bound on the spectral distortion of cyclotomic polynomials.

See List of Professors in Aaron Kirtland University(Washington University in St. Louis)

Aaron Kirtland FAQs

What is Aaron Kirtland's h-index at Washington University in St. Louis?

The h-index of Aaron Kirtland has been 3 since 2020 and 3 in total.

What are Aaron Kirtland's top articles?

The articles with the titles of

REBUS: A Robust Evaluation Benchmark of Understanding Symbols

An unstructured mesh approach to nonlinear noise reduction for coupled systems

Inverse Scaling: When Bigger Isn't Better

The polynomial learning with errors problem and the smearing condition

The Ring Learning With Errors Problem: Spectral Distortion

are the top articles of Aaron Kirtland at Washington University in St. Louis.

What is Aaron Kirtland's total number of citations?

Aaron Kirtland has 59 citations in total.

    academic-engine

    Useful Links