The following sections contain some background information on the algorithms used in VASP. They do not contain a complete reference to all the things implemented in VASP but try to give hints on the most important topics. You should really understand at least the ideas touched here -- but it might be still possible to get good results without understanding all of it.
For a basic outline of pseudopotential plane wave programs we refer to [6,7]. Ultrasoft pseudopotentials are explained in [8,9,10,18]. An excellent introduction to PP plane wave codes - albeit in German - can be found in the thesis of J. Furthmüller [11]. The best explanation of the algorithms found in VASP can be found in Ref. [13,14], these two papers give much more information than can be found in the following sections. And last but not least, you want might read the thesis of G. Kresse [12] (in German too) -- it contains a general discussion of PP including ultrasoft PP, and a discussion of the KS-functional and algorithms to calculate the KS-groundstate.