Abstract
Traditional information theoretic measures used for evaluating channel capacity and
lossy compression are defined via mutual information. For memoryless communication
channels and sources this measure has been successfully applied to compute the
operation capacity of channels and lossy compression of sources, respectively. For channels
with memory and feedback, and nonanticipative lossy compression of sources with
memory the valid information measure is the directed information defined via nonanticipative
conditional distributions. Directed information is also extensively utilized in
networks, communication for real-time stochastic control applications, and in biological
system analysis.
This thesis investigates the functional and topological properties of directed information
and two extremum problems arising from this information theoretic measure. The
first, is the extremum problem of nonanticipative rate distortion function of sources
with memory and the second, is the extremum problem of feedback capacity of channels
with memory and feedback. For these two extremum problems, existence of an optimal
solution is shown using the topology of weak convergence of probability distributions.
For the extremum problem of nonanticipative rate distortion function, applications in
zero-delay Joint Source-Channel Coding (JSCC) design based on average and excess
distortion probability, in bounding the Optimal Performance Theoretically Attainable
(OPTA) by noncausal and causal codes, and computing the Rate Loss (RL) of zerodelay
and causal codes with respect to noncausal codes are derived. For the extremum
problem of feedback capacity, sequential necessary and sufficient conditions are derived
and applied to time-varying channels with memory to establish recursive closed form expressions
of the optimal distributions, which maximize the finite-time horizon directed
information. In addition, the feedback capacity of several time-invariant channels with
memory is derived using the asymptotic properties of the optimal distributions of the
finite-time horizon directed information.