Geometric Inference in Machine Learning: Applications of Fisher Information for Model Selection and Other Statistical Applications
Date of Award
Doctor of Philosophy (PhD)
Computer Engineering and Sciences
Adrian M. Peter
Luis Daniel Otero
We consider the problem of model selection using the Minimum Description Length (MDL) criterion for distributions with parameters on the hypersphere. Model selection algorithms aim to find a compromise between goodness of fit and model complexity. Variables often considered for complexity penalties involve number of parameters, sample size and shape of the parameter space, with the penalty term often referred to as stochastic complexity. Because Laplace approximation techniques yield inaccurate results for curved spaces, existing criteria incorrectly penalize complexity. We demonstrate how the use of a constrained Laplace approximation on the hypersphere yields a novel complexity measure that more accurately reflects the geometry of these spherical parameters spaces. We refer to this modified model selection criterion as spherical MDL. As proof of concept, spherical MDL is used for bin selection in histogram density estimation, performing favorably against other model selection criteria. Furthermore, we consider the problem of identifying the most similar distribution from a constrained set to a given distribution. We measure similarity using a symmetric distance on the manifold governed by the Fisher information metric, with a smaller distance on the manifold indicating distributions that are more similar. For the most part, research into the geodesic problem on manifolds is limited to the paths between two known distribution. Allowing one or both of the endpoint distributions to belong to a constrained surface on the manifold requires the introduction of transversality conditions and the techniques from variational calculus. We show the efficacy of this approach by applying it to different manifolds and constraint surfaces, including Gaussian manifolds with the isotropic constraint surface.
Herntier, Trevor, "Geometric Inference in Machine Learning: Applications of Fisher Information for Model Selection and Other Statistical Applications" (2023). Theses and Dissertations. 1239.
Copyright held by author