Theoretical and Applied Data Science Lunch-n-learn: Large scale inference for correlation and partial correlation
Speaker:Dr. Yumov Qiu
Title: Large scale inference for correlation and partial correlation
Abstract: Correlation and partial correlations are commonly used to study the dependence structure among variables. Correlation measures the marginal dependence of variables and partial correlation measures their conditional dependence after controlling the effects of all other variables. We will first discuss testing for specific structures of large correlation and partial correlation matrices. We propose a multi-level thresholding test that is shown to be powerful in detecting sparse and weak signals. The proposed test has attractive detection boundary and attains the optimal minimax rate in the signal strength under different regimes of high dimensionality and the sparsity of the signal. Second, we will talk about multiple testing procedures to identify the nonzero partial correlations. The proposed procedure is adaptive to temporally dependent data and is able to control false discovery proportion. The potential applications of the proposed methods will also be discussed.