CIS 6200: Advanced Topics in Machine Learning

Course Description and Objectives

This course will cover selected topics in the emerging theory of large language models (LLMs). The goal is for you to learn how researchers have begun to study these systems mathematically: how to formulate precise questions and how to tackle them with the right abstractions and tools. We will work through core techniques and proofs in detail. Because this is a fast-moving area, the set of topics may evolve during the semester.

Syllabus and Readings

The balance among topics will depend on the pace of the course and on student interest. Lecture notes will be posted here as the semester progresses.

1. Overview of large language models (notes). How a language model is defined and used: tokens, next-token prediction, prompts, and generating a response.
2. Language generation and hallucination. What does it mean to learn to generate language from data that only ever shows valid text? We will ask what is possible to learn with this data, what we should demand of a generator beyond correctness, and why a model that fits its data well can still hallucinate.
3. Next-token prediction and pretraining. LLMs are trained to predict the next token in a sequence. We will ask what this objective measures and explore its success and pitfalls.
4. Transformer architecture and computation. Most state-of-the-art LLMs are based on Transformer architecures. To understand this architecture, we will ask what computations can a transformer represent, and how this depends on the depth of the model, the length of the sequence, etc.
5. Scaling laws. Loss falls predictably as model size, data, and compute grow. We will study these empirical laws, the compute-optimal tradeoff they imply between model size and data, and possible explanations for why they hold.
6. In-context learning. A pretrained model can learn a new task from a few examples in its prompt, with no change to the model. We will study different mechanisms for understanding how this is possible.
7. Reasoning, search, and verification. Given a trained model, instead of training it further, we can spend more computation by generating more tokens for each prompt: intermediate reasoning steps, many completions, or use a verifier that checks candidates. We will study when and why this inference-time computation helps.
8. Post-training and learning from feedback. How much can feedback change a model after pretraining? We study supervised fine-tuning on demonstrations, reward models learned from preference comparisons, and reinforcement learning against a reward model.

We will not cover training dynamics, optimization, mechanistic interpretability, systems engineering, diffusion, or safety. These are excellent topics for your projects.

Selected Readings

The following is a sample of the papers we will cover; a complete reading list for each topic will be posted as we go along.

• E. M. Gold. Language Identification in the Limit. Information and Control, 1967.
• D. Angluin. Inductive Inference of Formal Languages from Positive Data. Information and Control, 1980.
• J. Kleinberg, S. Mullainathan. Language Generation in the Limit. NeurIPS 2024.
• A. Kalai, S. Vempala. Calibrated Language Models Must Hallucinate. STOC 2024.
• C. E. Shannon. Prediction and Entropy of Printed English. Bell System Technical Journal, 1951.
• G. Bachmann, V. Nagarajan. The Pitfalls of Next-Token Prediction. ICML 2024.
• B. Liu, J. Ash, S. Goel, A. Krishnamurthy, C. Zhang. Transformers Learn Shortcuts to Automata. ICLR 2023.
• J. Kaplan et al. Scaling Laws for Neural Language Models. arXiv 2020.
• J. Hoffmann et al. Training Compute-Optimal Large Language Models. arXiv 2022.
• T. Brown et al. Language Models are Few-Shot Learners. NeurIPS 2020.
• S. M. Xie, A. Raghunathan, P. Liang, T. Ma. An Explanation of In-Context Learning as Implicit Bayesian Inference. ICLR 2022.
• C. Olsson et al. In-context Learning and Induction Heads. Transformer Circuits Thread, 2022.
• A. Bietti, V. Cabannes, D. Bouchacourt, H. Jegou, L. Bottou. Birth of a Transformer: A Memory Viewpoint. NeurIPS 2023.
• E. Edelman, N. Tsilivis, B. Edelman, E. Malach, S. Goel. The Evolution of Statistical Induction Heads: In-Context Learning Markov Chains. NeurIPS 2024.
• J. von Oswald et al. Transformers Learn In-Context by Gradient Descent. ICML 2023.
• J. Wei et al. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. NeurIPS 2022.
• W. Merrill, A. Sabharwal. The Expressive Power of Transformers with Chain of Thought. ICLR 2024.
• E. Malach. Auto-Regressive Next-Token Predictors are Universal Learners. ICML 2024.
• N. Joshi, G. Vardi, A. Block, S. Goel, Z. Li, T. Misiakiewicz, N. Srebro. A Theory of Learning with Autoregressive Chain of Thought. COLT 2025.
• K. Cobbe et al. Training Verifiers to Solve Math Word Problems. arXiv 2021.
• A. Huang, A. Block, Q. Liu, N. Jiang, A. Krishnamurthy, D. Foster. Is Best-of-N the Best of Them? Coverage, Scaling, and Optimality in Inference-Time Alignment. ICML 2025.
• D. Rohatgi, A. Shetty, D. Saless, Y. Li, A. Moitra, A. Risteski, D. Foster. Taming Imperfect Process Verifiers: A Sampling Perspective on Backtracking. ICLR 2026.
• P. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, D. Amodei. Deep Reinforcement Learning from Human Preferences. NeurIPS 2017.
• L. Ouyang et al. Training Language Models to Follow Instructions with Human Feedback. NeurIPS 2022.
• D. Foster, A. Block, D. Misra. Is Behavior Cloning All You Need? Understanding Horizon in Imitation Learning. NeurIPS 2024.
• R. Rafailov et al. Direct Preference Optimization: Your Language Model Is Secretly a Reward Model. NeurIPS 2023.
• D. Foster, Z. Mhammedi, D. Rohatgi. Is a Good Foundation Necessary for Efficient Reinforcement Learning?. COLT 2025.
• A. Huang, A. Block, D. Foster, D. Rohatgi, C. Zhang, M. Simchowitz, J. Ash, A. Krishnamurthy. Self-Improvement in Language Models: The Sharpening Mechanism. ICLR 2025.

Recommended Preparation

There are no formal prerequisites. You should be comfortable with probability, linear algebra, calculus, and fundamental machine learning concepts, at the level of CIS 5200 or an equivalent course, and you should have the mathematical maturity to read research papers. Familiarity with deep learning and transformers is helpful but not required. We hope to keep the course self-contained, teaching the tools we need from first principles, but the pace will be fast.

Class Format

Lectures will develop the core theoretical concepts, mostly on the board. Papers will provide context and examples rather than a substitute for the lectures. The course project will give you an opportunity to explore an open question through theoretical or empirical work.

Requirements and Grading

1. Homework (25%): One handwritten problem set based on the mathematics developed in lecture, followed by a short one-on-one discussion of your submission with me or a teaching assistant.

2. Course Project (65%): The project must involve original research connected to one or more topics covered in the course, and it can be theoretical, empirical, or a mix of the two. For example, you might prove a new result following up on the works we discuss, define a new model to study, simplify a complicated existing result in a non-trivial way, or carry out an experimental evaluation inspired to verify or perhaps dispute theoretical predictions. You can be ambitious, but aim for something you can get done in a semester.

   You may work individually or in groups up to 3. Note that expectations scale with the size of the team, and your proposed topic must be approved by me. You may build on your current research if the project is clearly connected to the course.

   The project will be evaluated based on:

   • Project proposal (5%): At most two pages of content, plus references. State the research question, explain why it is interesting, describe your approach, and discuss the most relevant prior work. Please be mindful of scope.
   • Mid-semester presentation (15%): A short presentation covering your preliminary results, approaches that did not work, and your plan for the rest of the semester.
   • Final presentation (25%): A presentation covering the problem, approach, main results, and conclusions. The format and time limit will depend on the number of projects.
   • Project final report (20%): A report in the style of a NeurIPS paper, with at most six pages of content, plus references and appendices. You do not need to use all six pages.
3. In-Class Participation (10%): Based on attendance and contributions to class discussions.

The best outcome of the project would be a manuscript suitable for submission to a top machine learning conference or learning theory conference.

Tentative Dates

These dates will be confirmed at the start of the semester. There is no class on Thursday, October 1 (fall break) or Thursday, November 26 (Thanksgiving). The last lecture is on Thursday, November 19; the final two class meetings are reserved for project presentations.

Policies

We will primarily use Ed Discussion for course communication. Please post questions that may be useful to other students publicly, and contact me with a private post when needed.

Collaboration

Collaboration on the homework is permitted and encouraged. You may discuss the problems with other students, but you must understand, write, and submit your own solutions; you may not look at anyone else’s written solutions while preparing yours, and you may not post partial or complete solutions publicly on Ed. If you collaborate, please list your collaborators on your submission.

The same principle applies to AI tools, which you may use for both the homework and the course project, including for concept clarification, brainstorming, working through problems, editing, and code assistance. You are responsible for checking the correctness of all generated material and must understand everything you submit. Each submission should include a statement describing how you used AI tools, and you may be asked to explain your reasoning or discuss any part of your submission.

Late Work

If extenuating circumstances affect your ability to submit the homework or a project component on time, please contact me privately on Ed as early as possible.

Wellness Statement

Your mental health and wellbeing are incredibly important to me, and I recognize the stresses, both academic and personal, that many of you may be under at Penn. I encourage you to come speak with me if any extenuating circumstances arise during the semester.

I also want to make sure you are aware of some of the mental health resources at Penn:

• Student Health and Counseling: the primary care and psychological services division of Wellness at Penn, where you can schedule counseling appointments.
• Mental Health at Penn: the university’s mental health hub.
• For urgent concerns, contact CAPS at 215-898-7021 for emergency help.

If you are a victim of, witness to, or otherwise affected by unacceptable behavior:

• In cases of sexual harassment or assault, contact DPS Special Services at 215-573-3333, or 511 from a campus phone. This is a confidential resource.
• To report bias incidents, contact the Penn Office of Diversity.
• For other violations of the Code of Student Conduct, contact the Office of Student Conduct.

DEI Statement

At Penn and in this course, we value and actively seek to include all students and their unique identities, including but not limited to ethnicity, gender, sexual orientation, country of origin, class, religion, and disability status. You belong here, and I am here to help you. If at any point you feel that the course is not providing an inclusive environment for you or your classmates, please let me know. I want the lectures, discussions, and office hours to be spaces where you feel comfortable learning and sharing your ideas.

Changes

I reserve the right to adjust the syllabus, including lecture content and homework and project deadlines; any changes will be announced as early as possible.