Logistics

Time/Location:

• Tuesdays 3:00pm-4:20pm in CS 301
• Office hours:
  Karthik Narasimhan: Mondays 4pm-5pm, Computer Science 422
  Xi Chen: Fridays 1:30 - 3:30pm, CS 003

How to contact us: Please use Piazza for all questions related to this IW seminar and to find announcements. For external queries, emergencies, or personal matters, you can use a private Piazza post visible only to Instructors.

Content

What is this seminar about?

Reinforcement learning has enjoyed a healthy resurgence in recent years, with super-human performance being demonstrated on domains like Go, Starcraft and Atari. This seminar aims to provide students the opportunity to learn the basics and acquire hands-on experience with RL techniques through a semester-long project. Types of projects include benchmarking state-of-the-art RL algorithms on various tasks, proposing new algorithms for more sample-efficient or generalizable RL, applying existing techniques to solve problems in other domains (e.g. computer vision, NLP, program synthesis, or even job scheduling!), or development of new evaluation frameworks, tasks and metrics for RL.
There are no prerequisites for this seminar beyond COS 217, COS 226 and one of COS 324 (Machine Learning), 484 (NLP), 485 (Neural Networks) or a similar machine learning course. You will be expected to implement RL algorithms, preferably using standard libraries like PyTorch, Tensorflow, OpenAI gym, etc. Students may work in pairs IF the project can be split so that each student has their own semester-size piece of the project. We will spend the first 1-2 meetings of the semester surveying standard and current RL algorithms, brainstorming ideas and developing project proposals. The remaining meetings will be used for project updates, formal student presentations, and discussions on how to perform background research, prepare a presentation and write a final paper.

Resources

Courses:

• Video lectures from David Silver's RL course
• CS 285 taught by Sergey Levine, UC Berkeley: Fall 2019

Readings:

Some helpful books to read up on background material:

• Richard Sutton and Andrew Barto Reinforcement Learning: An Introduction
• Ian Goodfellow and Yoshua Bengio and Aaron Courville Deep Learning
• Csaba Szepesvari Algorithms for Reinforcement Learning

Useful web resources:

• Introduction to PyTorch
• OpenAI gym
• OpenAI Baselines
• RLLib

Funding:

• You may apply for funding to support your IW project (deadline in Feb, exact date will be announced soon on the general IW page)

Schedule

Pre-class action items should be completed by 11:59pm on Mondays before class on Tuesdays. Please also keep track of general IW events and deadlines here.

Date	Topic	Pre-class Action items
Feb 4	Introduction	Fill out this form
Feb 6	Information meeting for all IW students (CS 105, 4:30-5:30pm)
Feb 7	Introduction to PyTorch
Feb 11	Brainstorming project ideas	Project ideas form
Feb 18	Project proposal presentations	Presentation in Google slides
Feb 25	Project updates	Update Google slides
Mar 3	Project updates	Update Google slides
Mar 8	Submit checkpoint form
Mar 10	Project updates	Update Google slides
Mar 17	No class (Spring recess)
Mar 24	Project updates	Update Google slides
Mar 31	Project updates, Attend "How to Give an IW talk" in CS 105 at 4:30pm	Update Google slides
Apr 7	Project updates	Update Google slides
Apr 14	Project updates, Attend "How to write an IW paper" in CS 105 at 4:30pm	Update Google slides
Apr 26	Submit slides for Oral Presentation
Apr 28	In-class oral presentations
May 12	Written final report due