EE514A/EE515A - Information Theory I/II - Spring 2012/Winter 2012 Quarter

Last updated: $Id: index.html 566 2012-05-16 01:30:32Z bilmes $

This page is located at http://j.ee.washington.edu/~bilmes/classes/ee515a_spring_2012/

Instructor:

Prof. Jeff A. Bilmes --- Email me
Office: 418 EE/CS Bldg., +1 206 221 5236
Office hours: TBD, EEB-418

TA (for IT-I EE514A only, no TA for IT-II EE515A):

Karthik N. Mohan Office: EEB-258
Office hours: Monday 3:30 PM - 5:00 PM @ AE 107 (Inside EE Office, 1st floor) Friday 4:30 PM - 6:00 PM @ EEB 258

Announcements II

Information I and II

Description: This course will cover the basics of information theory.

Information Theory I: entropy, mutual information, asymptotic equipartition properties, data compression to the entropy limit (source coding theorem), Huffman, Lempel-Ziv, convolutional codes, communication at the channel capacity limit (channel coding theorem), method of types, differential entropy, maximum entropy.

Information Theory II (EE515, Spring 2012) : ECC, turbo, LDPC and other codes, Kolmogorov complexity, spectral estimation, rate-distortion theory, alternating minimization for computation of RD curve and channel capacity, more on the Gaussian channel, network information theory, information geometry, and some recent results on use of polymatroids in information theory.

Additional topics throughout will include information theory as it is applicable to pattern recognition, natural language processing, computer science and complexity, biological science, and communications.

Course Format: Two two-hour lectures per week, WF 10:30-12:20, EEB-026

See the first lecture (lecture 19 from 3/28/2012) for more general information about the course

Homework IT-II

Homework must be done and submitted electronically via the following link here.
  1. Homework 2, due Wednesady May 16th, 11:45pm via our dropbox. Do book problems: 8.1, 8.8, 9.1, 9.2, 9.6, 10.5, 10.6, 13.5, 13.6, 14.3, 14.4, 14.5
  2. Homework 1, due Wednesday May 9th, at 11:00pm, was assigned in class, see pdf page 161 of this link

Lecture Slides II (EE515A, Spring 2012)

Lecture slides will be made available as they are being prepared --- they will probably appear here right before a given lecture, and they will be in PDF format (original source is latex). Note, that these slides are corrected after the lecture (and might also include some additional discussion we had during lecture). If you find bugs/typos in these slides, please email me. The slides are available as "slides" and also in 2-up form for printing.
Lec. # Slides Print Lecture Date Last Updated Contents
19 pdf pdf 3/28/12 3/28/12 Intro to class, differential entropy.
20 pdf pdf 3/30/12 3/30/12 differential entropy, Gaussian channel
21 pdf pdf 4/4/12 4/4/12 Gaussian channel, parallel Gaussian channels
22 pdf pdf 4/11/12 4/12/12 parallel Gaussian channels, colored noise, feeback
23 pdf pdf 4/13/12 4/13/12 parallel Gaussian ch, feedback, matrix ineq, begin rate distortion
24 pdf pdf 4/18/12 4/19/12 rate distortion
25 pdf pdf 4/20/12 4/24/12 rate distortion theorem
26 pdf pdf 4/25/12 4/25/12 computing rate distortion
27 pdf pdf 4/27/12 5/1/12 alternating minimization
28 pdf pdf 5/2/12 5/1/12 alternating minimization proofs
29 pdf pdf 5/4/12 5/7/12 Kolmogorov Complexity
30 pdf pdf 5/9/12 5/9/12 Kolmogorov Complexity, Lempel-Ziv
31 pdf pdf 5/11/12 5/15/12 Lempel-Ziv
32 pdf pdf 5/16/12 5/16/12 Finish Lempel-Ziv, start I-measures
Lec. # Slides Print Lecture Date Last Updated Contents

Actually Presented Lecture Slides, Spring, 2012

Lecture slides that were presented in class, along with all of the bugs and typos, my ink corrections of (perhaps some) of the bugs and typos, and any other little notes/discussions/drawings I drew on the slides during class. The above slides often contain more material than these as any discussions during class were added to the above after class. On the other hand, there might be a few hand-drawn figures in the below that I have not yet added to the above. Note: not all PDF readers can see the annotations in these slides (e.g., at least the Safari embedded reader on the iphone/ipad doesn't see them) --- the annotations were done with Adobe acrobat.
  1. Lecture 19 from 3/28/12.
  2. Lecture 20 from 3/30/12.
  3. Lecture 21 from 4/4/12.
  4. Lecture 22 from 4/11/12.
  5. Lecture 23 from 4/13/12.
  6. Lecture 24 from 4/18/12.
  7. Lecture 25 from 4/20/12.
  8. Lecture 26 from 4/25/12.
  9. Lecture 27 from 4/27/12.
  10. Lecture 28 from 5/2/12.
  11. Lecture 29 from 5/4/12.
  12. Lecture 30 from 5/9/12.
  13. Lecture 31 from 5/11/12.

Material from first half of course, IT I - EE514A

Announcements I

Homework IT-I

Homework must be done and submitted electronically via the following link here.
  1. Homework 6 Due next electronically Friday, Mar 9th, 5:00pm.
  2. Homework 5 Due next electronically Thursday, Mar 1st, 11:45pm. Homework 5 Solutions.
  3. Homework 4 Due next electronically Saturday, Feb 11th, 11:45pm. Homework 4 Solutions.
  4. Homework 3 Due next electronically Sunday, Feb 5th, 5:00pm. Homework 3 Solutions.
  5. Homework 2 Due next electronically Thursday, Jan 26th, 11:45pm. Homework 2 Solutions.
  6. Homework 1 Due next electronically Monday, Jan 16th, 11:45pm. Homework 1 Solutions.
  7. Homework 0 Due next electronically Monday, Jan 9th, 11:45pm. Homework 0 Solutions.

Lecture Slides I (EE514A, Winter 2012)

Lecture slides will be made available as they are being prepared --- they will probably appear here right before a given lecture, and they will be in PDF format (original source is latex). Note, that these slides are corrected after the lecture (and might also include some additional discussion we had during lecture). If you find bugs/typos in these slides, please email me. The slides are available as "slides" and also in 2-up form for printing.
Lec. # Slides Print Lecture Date Last Updated Contents
0 pdf pdf 1/3/12 1/5/12 Preliminaries, probability, convexity, Jensen.
1 pdf pdf 1/3/12 1/3/12 Introduction, information, entropy
2 pdf pdf 1/5/12 1/11/12 entropy, MI, KL-divergence
3 pdf pdf 1/10/12 1/11/12 KL-Divergence, Jensen, properties, Data Proc. Inequality, Thermodynamics
4 pdf pdf 1/12/12 2/13/12 Data Proc.\ Ineq., thermodynamics, Stats, Fano
5 pdf pdf 1/17/12 1/18/12 Fano, WLLN, Conv, AEP
6 pdf pdf 1/19/12 1/19/12 Seattle snow storm
6 pdf pdf 1/24/12 1/26/12 AEP, source coding
7 pdf pdf 1/26/12 1/26/12 types, source coding
8 pdf pdf 1/31/12 1/31/12 source coding, entropy rates
9 pdf pdf 2/2/12 2/6/12 HMMs, coding
10 pdf pdf 2/7/12 2/7/12 coding, Kraft, Huffman
11 pdf pdf 2/9/12 2/9/12 Huffman, Fano
12 pdf pdf 2/16/12 2/29/12 Arithmetic coding
13 pdf pdf 2/21/12 2/29/12 Channel capacity
14 pdf pdf 2/23/12 2/27/12 Towards Shannon's Thm.
15 pdf pdf 2/28/12 2/29/12 Shannon's Thm (first part)
16 pdf pdf 3/1/12 3/1/12 Shannon's 2nd Thm
17 pdf pdf 3/6/12 3/6/12 Shannon's 2nd Thm, Coding
18 pdf pdf 3/8/12 3/8/12 Hamming codes, intro diff entropy
Lec. # Slides Print Lecture Date Last Updated Contents

Actually Presented Lecture Slides, Winter, 2012

Lecture slides that were presented in class, along with all of the bugs and typos, my ink corrections of (perhaps some) of the bugs and typos, and any other little notes/discussions/drawings I drew on the slides during class. The above slides often contain more material than these as any discussions during class were added to the above after class. On the other hand, there might be a few hand-drawn figures in the below that I have not yet added to the above. Note: not all PDF readers can see the annotations in these slides (e.g., at least the Safari embedded reader on the iphone/ipad doesn't see them) --- the annotations were done with Adobe acrobat.
  1. Lecture 1 from 1/3/12.
  2. Lecture 2 from 1/5/12.
  3. Lecture 3 from 1/10/12.
  4. Lecture 4 from 1/12/12.
  5. Lecture 5 from 1/17/12.
  6. Lecture 6 from 1/24/12.
  7. Lecture 7 from 1/26/12.
  8. Lecture 8 from 1/31/12.
  9. Lecture 9 from 2/2/12.
  10. Lecture 10 from 2/7/12.
  11. Lecture 11 from 2/9/12.
  12. Lecture 12 from 2/16/12.
  13. Lecture 13 from 2/21/12.
  14. Lecture 14 from 2/23/12.
  15. Lecture 15 from 2/28/12.
  16. Lecture 16 from 3/1/12.
  17. Lecture 17 from 3/6/12.
  18. Lecture 18 from 3/8/12.

Discussion Board

You can post questions, discussion topics, or general information at this link.

Relevant Books

See lecture 1 slides for our text and relevant other texts.

Important Dates/Exceptions

Alternative Contact

If you must, you can send me anonymous email

Go to my home page Email me