# EE514A/EE515A - Information Theory I/II - Fall 2013/Winter 2014 Quarter - Fall Quarter, 2013

Last updated: $Id: index.html 1017 2013-12-03 21:58:53Z bilmes $

This page is located at http://j.ee.washington.edu/~bilmes/classes/ee514a_fall_2013/.

# Instructor:

**Prof. Jeff A. Bilmes**--- Email me

Office: 418 EE/CS Bldg., +1 206 221 5236

Office hours: Tuesdays TBD, EEB-418 (+ online)

### TA (for IT-I)

:**Jounsup Park**Office: EEB-361

Office hours: Monday 13:00-17:00 Wednesday 10:00-12:00

### Time/Location

Class is held: Tu/Th 2:30-4:20 PAC 297 (this is across campus from EE/CSE so give yourself a bit of time to get there, and read about the benefits of walking)# Announcements

- Sticky: Most of the announcements in our class are going to be posted on canvas, and can be found at this link.
- Homework 7 is now out, due Friday Nov 29th, at 11:45pm electronically via our dropbox.
- Homework 6 is now out, due Thursday Nov 21st, at 11:45pm electronically via our dropbox.
- Homework 5 is now out, due Thursday Nov 14th, at 11:45pm electronically via our dropbox.
- Homework 4 is now out, due Tuesday Oct 29nd, at 11:45pm electronically via our dropbox.
- Solutions to HW2 are now posted on canvas, see here.
- Homework 3 is now out, due Tuesday Oct 22nd, at 11:45pm electronically via our dropbox.
- Homework 2 is now out, due Tuesday Oct 15th, at 11:45pm electronically via our dropbox.
- Homework 1 is now out, due Tuesday Oct 8th, at 11:45pm electronically via our dropbox.
- Homework 0 is now out, due Tuesday Oct 1st, at 11:45pm electronically via our dropbox.
- (Thursday September 26th, 2013) Welcome to the class.

# Information I and II

Description: This course will cover the basics of information theory.

Information Theory I: entropy, mutual information, asymptotic equipartition properties, data compression to the entropy limit (source coding theorem), Huffman, Lempel-Ziv, convolutional codes, communication at the channel capacity limit (channel coding theorem), method of types, differential entropy, maximum entropy.

Information Theory II (EE515, Winter 2014) : ECC, turbo, LDPC and other codes, Kolmogorov complexity, spectral estimation, rate-distortion theory, alternating minimization for computation of RD curve and channel capacity, more on the Gaussian channel, network information theory, information geometry, and some recent results on use of polymatroids in information theory.

Additional topics throughout will include information theory as it is applicable to pattern recognition, natural language processing, computer science and complexity, biological science, and communications.

# Homework

Homework must be done and submitted electronically via the following link https://canvas.uw.edu/courses/847774/assignments.# Lecture Slides

Lecture slides will be made available as they are being prepared --- they will probably appear soon before a given lecture, and they will be in PDF format (original source is latex). Note, that these slides are corrected after the lecture (and might also include some additional discussion we had during lecture). If you find bugs/typos in these slides, please email me. The slides are available as "presentation slides" and also in (mostly the same content) 2-up form for printing. b b b b b b b b b bLec. # | Slides | Lecture Date | Last Updated | Contents | |

0 | -- | -- | Preliminaries, probability, convexity, Jensen. | ||

1 | 9/26/13 | 10/1/13 | Introduction, information, entropy | ||

2 | 10/1/13 | 10/1/13 | Entropy and properties, Mutual Information, | ||

3 | 10/3/13 | 10/3/13 | KL-Divergence, Concavity and Jensen, | ||

4 | 10/8/13 | 10/8/13 | Data Proc. Inequality, Entropy/Thermo, Suff. Stats, Fano, Modes of Conv., | ||

5 | 10/10/13 | 10/10/13 | Asymptotic Equipartition Property (AEP), Data Compression | ||

6 | 10/15/13 | 10/15/13 | AEP, Compression, Method of Types, | ||

7 | 10/17/13 | 10/17/13 | Method of Types, U. Coding., Stoc. Processes, Entropy rates, HMMs | ||

8 | 10/22/13 | 10/22/13 | Entropy rates, HMMs, Coding, Kraft, | ||

9 | 10/24/13 | 10/24/13 | Kraft, Shannon Codes, | ||

10 | 10/28/13 | 10/28/13 | Huffman, Shannon/Fano/Elias | ||

11 | 10/29/13 | 10/29/13 | Shannon Games, Arith. Coding | ||

- | - | - | 10/31/13 | 10/31/13 | Midterm, in class |

12 | 11/7/13 | 11/7/13 | Arith. Coding, Channel Capacity | ||

13 | 11/12/13 | 11/12/13 | Channel Capacity, Joint AEP, Shannon's 2nd theorem. | ||

14 | 11/14/13 | 11/14/13 | Channel Capacity | ||

15 | 11/19/13 | 11/19/13 | Shannon's 2nd theorem, converse, zero error codes, feedback | ||

16 | 11/21/13 | 11/21/13 | The joint source/channel coding theorem, coding methods, hamming coding, differential entropy | ||

17 | 11/26/13 | 11/26/13 | hamming coding, differential entropy | ||

18 | 12/3/13 | 12/3/13 | differential entropy, Gaussian channel | ||

Lec. # | Slides | Lecture Date | Last Updated | Contents |

# Actually Presented Lecture Slides and Videos, Fall, 2013

Lecture slides that were presented in class, along with all of the bugs and typos, my ink corrections of (perhaps some) of the bugs and typos, and any other little notes/discussions/drawings I drew on the slides during class. The above slides often contain more material than these as any discussions during class were added to the above after class. On the other hand, there might be a few hand-drawn figures in the below that I have not yet added to the above. Annotations were done using Goodnotes on the ipad and should be viewable by any respectable PDF reader.- Lecture 1 from 9/26/13.
- Lecture 2 from 10/1/13.
- Lecture 3 from 10/3/13.
- Lecture 4 from 10/8/13.
- Lecture 5 from 10/10/13.
- Lecture 6 from 10/15/13.
- Lecture 7 from 10/17/13.
- Lecture 8 from 10/22/13.
- Lecture 9 from 10/24/13.
- Lecture 10 from 10/28/13.
- Lecture 11 from 10/29/13.
- Lecture 12 from 11/7/13.
- Lecture 13 from 11/12/13.
- Lecture 14 from 11/14/13.
- Lecture 15 from 11/19/13.
- Lecture 16 from 11/21/13.
- Lecture 17 from 11/26/13.
- Lecture 18 from 12/7/13.

Lec. # | Presented Slides | Video Lecture |

Lecture 1 | ||

Lecture 2 | ||

Lecture 3 | ||

Lecture 4 | ||

Lecture 5 | ||

Lecture 6 | ||

Lecture 7 | ||

Lecture 8 | ||

Lecture 9 | ||

Lecture 10a | ||

Lecture 10b | ||

Lecture 11 | ||

Lecture 12 | ||

Lecture 13 | ||

Lecture 14 | ||

Lecture 15 | ||

Lecture 16 | ||

Lecture 17 | ||

Lecture 18 |

# Discussion Board

You can post questions, discussion topics, or general information at this link.

# Relevant Books

There are many books available that discuss some the material that we are covering in this course. See the end of the lecture slides for books/papers that are relevant to each specific lecture, and see lecture1.pdf for a description of our book (Cover and Thomas) and other books/papers relevant to this class.- See lecture 1 slides for our text and relevant other texts.

# Important Dates/Exceptions (also see academic calendar )

- Tuesday Nov 5th, 2013 - youtube video lecture day (no in-person instruction)
- Thursday Dec 5th, 2013 - youtube video lecture day (no in-person instruction)

# Alternative Contact

If you must, you can send me or the TA anonymous email