The aim of the present paper is to characterize the spectral representation of Gaussian semimartingales. That is, we provide necessary and sufficient conditions on the kernel $K$ for $(X_t)_{t\geq 0}=(\int K_t(s)\,dN_s)_{t\geq 0}$ to be a semimartingale. Here, $N$ denotes an independently scattered Gaussian random measure on a general space $S$. We study the semimartingale property of $(X_t)_{t\geq 0}$ in three different filtrations. First the $(\mathcal{F}^{X}_t)_{t\geq 0}$-semimartingale property is considered and afterwards the $(\mathcal{F}^{X,\infty}_t)_{t\geq 0}$-semimartingale property is treated in the case where $(X_t)_{t\in\mathbb{R}}$ is a moving average process and $\mathcal{F}^{X,\infty}_t=\sigma(X_s:s\in (-\infty,t])$. Finally we study a generalization of Gaussian Volterra processes. In particular we provide necessary and sufficient conditions on $K$ for the Gaussian Volterra process $(\int_{-\infty}^t K_t(s)\,dW_s)_{t\geq 0}$ to be an $(\mathcal{F}^{W,\infty}_t)_{t\geq 0}$-semimartingale ($(W_t)_{t\in\mathbb{R}}$ denotes a Wiener process). Hereby we generalize a result of Knight (1992), Cherny (2001) and Cheridito (2004) to the non-stationary case.