We improve the existing achievable rate regions for causal and for zero-delay source coding of stationary Gaussian sources for mean squared error (MSE) distortion. First, we define the information-theoretic causal rate-distortion function (RDF), Rcit(D). In order to analyze Rcit(D), we introduce Rcit(D), the information theoretic causal RDF when reconstruction error is jointly stationary with the source. Based upon Rcit(D), we derive four closed form upper bounds to the gap between Rcit(D) and Shannon's RDF, two of them strictly smaller than 0.5 bits/sample at all rates. We then show that Rcit(D) can be realized by an AWGN channel surrounded by a unique set of causal pre-, post-, and feedback filters. We show that finding such filters constitutes a convex optimization problem and propose an iterative procedure to solve it. Finally, we build upon Rcit(D) to improve existing bounds on the optimal performance attainable by causal and zero-delay codes. © 2010 IEEE.