|
·
Announcements
1. Homework
#3 due date is extended to March. 28.
2. Homework
#4 is posted.
·
Lecture
Mondays, Wednesdays, and Fridays, 2:10 pm – 3:00 pm, Beyer 1308
- Instructor
Guiping Hu
3033 Black Engineering Building
515-294-8638
gphu at iastate.edu
http://gphu.public.iastate.edu/
- Office Hours
By appointment
- Textbook
Martin Puterman, Markov Decision Processes,
2006.
Reference: Dimitris Bertsimas, Dynamic
Programming and Optimal Control, Vol I,
2005.
- Lecture
Notes
- Syllabus
|
|
Week
|
Date
|
Lecture
|
Homework
|
Notes
|
|
1
|
Jan. 9
|
Introduction to MDP
|
|
|
|
Jan.
11
|
Introduction
to MDP
|
|
|
|
Jan. 13
|
Examples
|
|
|
|
2
|
Jan.
16
|
|
|
NO CLASS
|
|
Jan. 18
|
MDP Model Formulation
|
Homework #1: 2.2, 2.5, 2.8, 3.2, 3.4, 3.9, 3.16
and 3.26
|
|
|
Jan.
20
|
MDP
Model Formulation
|
|
|
|
3
|
Jan. 23
|
MDP Applications
|
|
|
|
Jan.
25
|
MDP
Applications
|
|
|
|
Jan. 27
|
Finite Horizon MDP
|
|
|
|
4
|
Jan.
30
|
Finite
Horizon MDP
|
|
|
|
Feb. 1
|
Finite Horizon MDP
|
Homework
#2: 3.17, 4.1, 4.5, 4.6, 4.16, 4.20, 4.21 and 4.31
|
Homework
#1 is due
|
|
Feb.
3
|
Finite
Horizon MDP
|
|
|
|
5
|
Feb. 6
|
Finite Horizon MDP
|
|
|
|
Feb.
8
|
Project
#1 Presentations
|
|
Leilei, Yihua
|
|
Feb. 10
|
Project #1
Presentations
|
|
Bokan, Minhua
|
|
6
|
Feb.
13
|
Finite
Horizon MDP
|
|
|
|
Feb. 15
|
Finite Horizon MDP
|
|
|
|
Feb.
17
|
Finite
Horizon MDP
|
|
Homework
#2 is due
|
|
7
|
Feb. 20
|
Infinite Horizon MDP
|
|
|
|
Feb.
22
|
Infinite
Horizon MDP
|
|
|
|
Feb. 24
|
Discounted MDP
|
|
|
|
8
|
Feb.
27
|
Discounted
MDP
|
|
|
|
Feb. 29
|
Midterm Exam Review
|
|
|
|
Mar.
2
|
|
Homework
#3: 4.33a, 5.2, 5.12, 6.1a, 6.50a, 6.63a, 6.3 and 6.6
|
Midterm Exam due
|
|
9
|
Mar. 5
|
Value Iteration
|
|
|
|
Mar.
7
|
Value
Iteration
|
|
|
|
Mar. 9
|
Value Iteration
|
|
Proposal for Project 2
Due
|
|
10
|
Mar.
12
|
Spring
Break
|
|
NO CLASS
|
|
Mar. 14
|
Spring Break
|
|
NO CLASS
|
|
Mar.
16
|
Spring
Break
|
|
NO CLASS
|
|
11
|
Mar. 19
|
Proposal Presentation
on Project 2
|
|
|
|
Mar.
21
|
Value
Iteration
|
|
|
|
Mar. 23
|
Value Iteration
|
|
|
|
12
|
Mar.
26
|
|
|
CLASS CANCELLED
|
|
Mar. 28
|
Policy Iteration
|
Homework
#4: 6.15 a&b, 6.1b,c,d&e, 6.16,
6.19b&c,
and 6.20
|
Homework #3 is due
|
|
Mar.
30
|
Policy
Iteration
|
|
|
|
13
|
Apr. 2
|
LP for MDP
|
|
|
|
Apr.
4
|
LP
for MDP
|
|
|
|
Apr. 6
|
LP for MDP
|
|
|
|
14
|
Apr.
9
|
Structured
Policy
|
|
|
|
Apr. 11
|
Structured Policy
|
|
|
|
Apr.
13
|
Structured
Policy
|
|
Homework
#4 is due
|
|
15
|
Apr. 16
|
MDP Applications
|
|
|
|
Apr.
18
|
Final
Exam Review
|
|
|
|
Apr. 20
|
|
|
Final Exam due
|
|
16
|
Apr.
23
|
Project
2 Presentations
|
|
|
|
Apr. 25
|
Project 2
Presentations
|
|
Project #2 Report due 5pm
|
|
Apr.
27
|
|
|
|
|