# EE515A - Information Theory II, Winter Quarter, 2014

Last updated: $Id: index.html 1220 2014-03-13 19:44:47Z bilmes $

This is part II of a two quarter sequence on information theory. This page is located at http://j.ee.washington.edu/~bilmes/classes/ee515a_winter_2014/.

The web page for part I of the class held last quarter is located at http://j.ee.washington.edu/~bilmes/classes/ee514a_fall_2013/.

# Instructor:

**Prof. Jeff A. Bilmes**--- Email me

Office: 418 EE/CS Bldg., +1 206 221 5236

Office hours: TBD, EEB-418 (+ online)

### TA (for IT-II)

**Unfortunately, no TA. Sorry.**

Office: N/A

Office hours: N/A

### Time/Location

Class is held: Mon/Wed 1:30-3:20, Loew Hall, Room 102.# Announcements

- Sticky: Most of the announcements in our class are going to be posted on canvas, and can be found at this link.
- (Wed Jan 15th, 2014) HW1 is out, due 1/27/14 11:45pm. See canvas page for actual assignment, and to get solutions once they are posted.
- (Monday Jan 6th, 2014) Welcome to the class.

# Information I and II

Description: This course will cover the basics of information theory.

Information Theory I: entropy, mutual information, asymptotic equipartition properties, data compression to the entropy limit (source coding theorem), Huffman, Lempel-Ziv, convolutional codes, communication at the channel capacity limit (channel coding theorem), method of types, differential entropy, maximum entropy. IT-I was held fall quarater, 2013 and you can find this class here

Information Theory II (EE515, Winter 2014) : ECC, turbo, LDPC and other codes, Kolmogorov complexity, spectral estimation, rate-distortion theory, alternating minimization for computation of RD curve and channel capacity, more on the Gaussian channel, network information theory, information geometry, and some recent results on use of polymatroids in information theory.

Additional topics throughout will include information theory as it is applicable to pattern recognition, natural language processing, computer science and complexity, biological science, and communications.

# Homework

Homework must be done and submitted electronically via the following link https://canvas.uw.edu/courses/880971/assignments.# Lecture Slides

Lecture slides will be made available as they are being prepared --- they will probably appear soon before a given lecture, and they will be in PDF format (original source is latex). Note, that these slides are corrected after the lecture (and might also include some additional discussion we had during lecture). If you find bugs/typos in these slides, please email me. The slides are available as "presentation slides" and also in (mostly the same content) 2-up form for printing.Lec. # | Slides | Lecture Date | Last Updated | Contents | |

19 | 1/6/2014 | 1/6/2014 | Logistics, Gaussian Channel | ||

20 | 1/8/2014 | 1/8/2014 | Gaussian Channel, band limitation, parallel channels, optimization and duality | ||

21 | 1/13/2014 | 1/13/2014 | parallel channels, colored noise, feedback, matrix inequalities | ||

22 | 1/15/2014 | 1/15/2014 | matrix inequalities, rate distortion | ||

23 | 1/22/2014 | 1/22/2014 | rate distortion for Bernoulli, Gaussian, and Multiple Gaussians with unequal noise | ||

24 | 1/27/2014 | 1/27/2014 | main rate distortion theorem, geometry | ||

25 | 1/29/2014 | 1/29/2014 | computing R(D) | ||

26 | 2/3/2014 | 2/3/2014 | computing R(D), alternating minimization | ||

27 | 2/5/2014 | 2/5/2014 | Kolmogorov complexity | ||

28 | 2/10/2014 | 2/10/2014 | algorithmic randomness, Universal Prob., | ||

29 | 2/12/2014 | 2/12/2014 | Universal Compression, LZ compression | ||

30 | 2/19/2014 | 2/19/2014 | LZ compression | ||

31 | 2/24/2014 | 2/24/2014 | Info measures | ||

32 | 2/26/2014 | 2/26/2014 | Info measures, Info inequalities | ||

33 | 3/3/2014 | 3/3/2014 | Info inequalities, start NIT | ||

34 | 3/5/2014 | 3/5/2014 | NIT, MAC | ||

35 | 3/10/2014 | 3/10/2014 | MAC, Gaussian MAC | ||

36 | 3/12/2014 | 3/12/2014 | Slepian-Wolf, Broadcast, General |

# Actually Presented Lecture Slides and Videos, Winter, 2014

Lecture slides that were presented in class, along with all of the bugs and typos, my ink corrections of (perhaps some) of the bugs and typos, and any other little notes/discussions/drawings I drew on the slides during class. The above slides often contain more material than these as any discussions during class were added to the above after class. On the other hand, there might be a few hand-drawn figures in the below that I have not yet added to the above. Annotations were done using Goodnotes on the ipad and should be viewable by any respectable PDF reader.Lec. # | Date | Presented Slides | Video Lecture Link |

Lecture 19 | 1/6/2014 | ||

Lecture 20 | 1/8/2014 | ||

Lecture 21 | 1/13/2014 | ||

Lecture 22 | 1/15/2014 | ||

Lecture 23 | 1/22/2014 | ||

Lecture 24 | 1/27/2014 | ||

Lecture 25 | 1/29/2014 | ||

Lecture 26 | 2/3/2014 | ||

Lecture 27 | 2/5/2014 | ||

Lecture 28 | 2/10/2014 | ||

Lecture 29 | 2/12/2014 | ||

Lecture 30 | 2/19/2014 | ||

Lecture 31 | 2/24/2014 | ||

Lecture 32 | 2/26/2014 | ||

Lecture 33 | 3/3/2014 | ||

Lecture 34 | 3/5/2014 | ||

Lecture 35 | 3/10/2014 | ||

Lecture 36 | 3/12/2014 |

# Discussion Board

You can post questions, discussion topics, or general information at this link.

# Relevant Books

There are many books available that discuss some the material that we are covering in this course. See the end of the lecture slides for books/papers that are relevant to each specific lecture, and see lecture1.pdf for a description of our book (Cover and Thomas) and other books/papers relevant to this class.- See lecture 19 slides for our text and relevant other texts.

# Important Dates/Exceptions (also see academic calendar )

- Monday, 1/20, no class (holiday).
- Monday, 2/17, no class (holiday).
- Monday, 3/17, final presentations in class.

# Alternative Contact

If you must, you can send me or the TA anonymous email