EE515A - Information Theory II, Winter Quarter, 2014

Last updated: $Id: index.html 1220 2014-03-13 19:44:47Z bilmes $

This is part II of a two quarter sequence on information theory. This page is located at

The web page for part I of the class held last quarter is located at


Prof. Jeff A. Bilmes --- Email me
Office: 418 EE/CS Bldg., +1 206 221 5236
Office hours: TBD, EEB-418 (+ online)

TA (for IT-II)

Unfortunately, no TA. Sorry.
Office: N/A
Office hours: N/A


Class is held: Mon/Wed 1:30-3:20, Loew Hall, Room 102.


Information I and II

Description: This course will cover the basics of information theory.

Information Theory I: entropy, mutual information, asymptotic equipartition properties, data compression to the entropy limit (source coding theorem), Huffman, Lempel-Ziv, convolutional codes, communication at the channel capacity limit (channel coding theorem), method of types, differential entropy, maximum entropy. IT-I was held fall quarater, 2013 and you can find this class here

Information Theory II (EE515, Winter 2014) : ECC, turbo, LDPC and other codes, Kolmogorov complexity, spectral estimation, rate-distortion theory, alternating minimization for computation of RD curve and channel capacity, more on the Gaussian channel, network information theory, information geometry, and some recent results on use of polymatroids in information theory.

Additional topics throughout will include information theory as it is applicable to pattern recognition, natural language processing, computer science and complexity, biological science, and communications.


Homework must be done and submitted electronically via the following link

Lecture Slides

Lecture slides will be made available as they are being prepared --- they will probably appear soon before a given lecture, and they will be in PDF format (original source is latex). Note, that these slides are corrected after the lecture (and might also include some additional discussion we had during lecture). If you find bugs/typos in these slides, please email me. The slides are available as "presentation slides" and also in (mostly the same content) 2-up form for printing.
Lec. # Slides Print Lecture Date Last Updated Contents
19 pdf pdf 1/6/2014 1/6/2014 Logistics, Gaussian Channel
20 pdf pdf 1/8/2014 1/8/2014 Gaussian Channel, band limitation, parallel channels, optimization and duality
21 pdf pdf 1/13/2014 1/13/2014 parallel channels, colored noise, feedback, matrix inequalities
22 pdf pdf 1/15/2014 1/15/2014 matrix inequalities, rate distortion
23 pdf pdf 1/22/2014 1/22/2014 rate distortion for Bernoulli, Gaussian, and Multiple Gaussians with unequal noise
24 pdf pdf 1/27/2014 1/27/2014 main rate distortion theorem, geometry
25 pdf pdf 1/29/2014 1/29/2014 computing R(D)
26 pdf pdf 2/3/2014 2/3/2014 computing R(D), alternating minimization
27 pdf pdf 2/5/2014 2/5/2014 Kolmogorov complexity
28 pdf pdf 2/10/2014 2/10/2014 algorithmic randomness, Universal Prob.,
29 pdf pdf 2/12/2014 2/12/2014 Universal Compression, LZ compression
30 pdf pdf 2/19/2014 2/19/2014 LZ compression
31 pdf pdf 2/24/2014 2/24/2014 Info measures
32 pdf pdf 2/26/2014 2/26/2014 Info measures, Info inequalities
33 pdf pdf 3/3/2014 3/3/2014 Info inequalities, start NIT
34 pdf pdf 3/5/2014 3/5/2014 NIT, MAC
35 pdf pdf 3/10/2014 3/10/2014 MAC, Gaussian MAC
36 pdf pdf 3/12/2014 3/12/2014 Slepian-Wolf, Broadcast, General

Actually Presented Lecture Slides and Videos, Winter, 2014

Lecture slides that were presented in class, along with all of the bugs and typos, my ink corrections of (perhaps some) of the bugs and typos, and any other little notes/discussions/drawings I drew on the slides during class. The above slides often contain more material than these as any discussions during class were added to the above after class. On the other hand, there might be a few hand-drawn figures in the below that I have not yet added to the above. Annotations were done using Goodnotes on the ipad and should be viewable by any respectable PDF reader.
Lec. # Date Presented Slides Video Lecture Link
Lecture 19 1/6/2014 pdf
Lecture 20 1/8/2014 pdf
Lecture 21 1/13/2014 pdf
Lecture 22 1/15/2014 pdf
Lecture 23 1/22/2014 pdf
Lecture 24 1/27/2014 pdf
Lecture 25 1/29/2014 pdf
Lecture 26 2/3/2014 pdf
Lecture 27 2/5/2014 pdf
Lecture 28 2/10/2014 pdf
Lecture 29 2/12/2014 pdf
Lecture 30 2/19/2014 pdf
Lecture 31 2/24/2014 pdf
Lecture 32 2/26/2014 pdf
Lecture 33 3/3/2014 pdf
Lecture 34 3/5/2014 pdf
Lecture 35 3/10/2014 pdf
Lecture 36 3/12/2014 pdf

Discussion Board

You can post questions, discussion topics, or general information at this link.

Relevant Books

There are many books available that discuss some the material that we are covering in this course. See the end of the lecture slides for books/papers that are relevant to each specific lecture, and see lecture1.pdf for a description of our book (Cover and Thomas) and other books/papers relevant to this class.

Important Dates/Exceptions (also see academic calendar )

Alternative Contact

If you must, you can send me or the TA anonymous email