EE514A/EE515A - Information Theory I/II - Fall 2013/Winter 2014 Quarter - Fall Quarter, 2013

Last updated: $Id: index.html 1017 2013-12-03 21:58:53Z bilmes $

This page is located at


Prof. Jeff A. Bilmes --- Email me
Office: 418 EE/CS Bldg., +1 206 221 5236
Office hours: Tuesdays TBD, EEB-418 (+ online)

TA (for IT-I)

: Jounsup Park Office: EEB-361
Office hours: Monday 13:00-17:00 Wednesday 10:00-12:00


Class is held: Tu/Th 2:30-4:20 PAC 297 (this is across campus from EE/CSE so give yourself a bit of time to get there, and read about the benefits of walking)


Information I and II

Description: This course will cover the basics of information theory.

Information Theory I: entropy, mutual information, asymptotic equipartition properties, data compression to the entropy limit (source coding theorem), Huffman, Lempel-Ziv, convolutional codes, communication at the channel capacity limit (channel coding theorem), method of types, differential entropy, maximum entropy.

Information Theory II (EE515, Winter 2014) : ECC, turbo, LDPC and other codes, Kolmogorov complexity, spectral estimation, rate-distortion theory, alternating minimization for computation of RD curve and channel capacity, more on the Gaussian channel, network information theory, information geometry, and some recent results on use of polymatroids in information theory.

Additional topics throughout will include information theory as it is applicable to pattern recognition, natural language processing, computer science and complexity, biological science, and communications.


Homework must be done and submitted electronically via the following link

Lecture Slides

Lecture slides will be made available as they are being prepared --- they will probably appear soon before a given lecture, and they will be in PDF format (original source is latex). Note, that these slides are corrected after the lecture (and might also include some additional discussion we had during lecture). If you find bugs/typos in these slides, please email me. The slides are available as "presentation slides" and also in (mostly the same content) 2-up form for printing. b b b b b b b b b b
Lec. # Slides Print Lecture Date Last Updated Contents
0 pdf pdf ---- Preliminaries, probability, convexity, Jensen.
1 pdf pdf 9/26/13 10/1/13 Introduction, information, entropy
2 pdf pdf 10/1/1310/1/13 Entropy and properties, Mutual Information,
3 pdf pdf 10/3/13 10/3/13 KL-Divergence, Concavity and Jensen,
4 pdf pdf 10/8/1310/8/13 Data Proc. Inequality, Entropy/Thermo, Suff. Stats, Fano, Modes of Conv.,
5 pdf pdf 10/10/13 10/10/13 Asymptotic Equipartition Property (AEP), Data Compression
6 pdf pdf 10/15/1310/15/13 AEP, Compression, Method of Types,
7 pdf pdf 10/17/13 10/17/13 Method of Types, U. Coding., Stoc. Processes, Entropy rates, HMMs
8 pdf pdf 10/22/1310/22/13 Entropy rates, HMMs, Coding, Kraft,
9 pdf pdf 10/24/13 10/24/13 Kraft, Shannon Codes,
10 pdf pdf 10/28/1310/28/13 Huffman, Shannon/Fano/Elias
11 pdf pdf 10/29/13 10/29/13 Shannon Games, Arith. Coding
- - - 10/31/13 10/31/13 Midterm, in class
12 pdf pdf 11/7/1311/7/13 Arith. Coding, Channel Capacity
13 pdf pdf 11/12/13 11/12/13 Channel Capacity, Joint AEP, Shannon's 2nd theorem.
14 pdf pdf 11/14/1311/14/13 Channel Capacity
15 pdf pdf 11/19/13 11/19/13 Shannon's 2nd theorem, converse, zero error codes, feedback
16 pdf pdf 11/21/1311/21/13 The joint source/channel coding theorem, coding methods, hamming coding, differential entropy
17 pdf pdf 11/26/13 11/26/13 hamming coding, differential entropy
18 pdf pdf 12/3/1312/3/13 differential entropy, Gaussian channel
Lec. # Slides Print Lecture Date Last Updated Contents

Actually Presented Lecture Slides and Videos, Fall, 2013

Lecture slides that were presented in class, along with all of the bugs and typos, my ink corrections of (perhaps some) of the bugs and typos, and any other little notes/discussions/drawings I drew on the slides during class. The above slides often contain more material than these as any discussions during class were added to the above after class. On the other hand, there might be a few hand-drawn figures in the below that I have not yet added to the above. Annotations were done using Goodnotes on the ipad and should be viewable by any respectable PDF reader.
  1. Lecture 1 from 9/26/13.
  2. Lecture 2 from 10/1/13.
  3. Lecture 3 from 10/3/13.
  4. Lecture 4 from 10/8/13.
  5. Lecture 5 from 10/10/13.
  6. Lecture 6 from 10/15/13.
  7. Lecture 7 from 10/17/13.
  8. Lecture 8 from 10/22/13.
  9. Lecture 9 from 10/24/13.
  10. Lecture 10 from 10/28/13.
  11. Lecture 11 from 10/29/13.
  12. Lecture 12 from 11/7/13.
  13. Lecture 13 from 11/12/13.
  14. Lecture 14 from 11/14/13.
  15. Lecture 15 from 11/19/13.
  16. Lecture 16 from 11/21/13.
  17. Lecture 17 from 11/26/13.
  18. Lecture 18 from 12/7/13.
Lec. # Presented Slides Video Lecture
Lecture 1 pdf
Lecture 2 pdf
Lecture 3 pdf
Lecture 4 pdf
Lecture 5 pdf
Lecture 6 pdf
Lecture 7 pdf
Lecture 8 pdf
Lecture 9 pdf
Lecture 10a pdf
Lecture 10b pdf
Lecture 11 pdf
Lecture 12 pdf
Lecture 13 pdf
Lecture 14 pdf
Lecture 15 pdf
Lecture 16 pdf
Lecture 17 pdf
Lecture 18 pdf

Discussion Board

You can post questions, discussion topics, or general information at this link.

Relevant Books

There are many books available that discuss some the material that we are covering in this course. See the end of the lecture slides for books/papers that are relevant to each specific lecture, and see lecture1.pdf for a description of our book (Cover and Thomas) and other books/papers relevant to this class.

Important Dates/Exceptions (also see academic calendar )

Alternative Contact

If you must, you can send me or the TA anonymous email