GAMESS references
There are two literature references for GAMESS itself,
"General Atomic and Molecular Electronic Structure System" M.W.Schmidt, K.K.Baldridge, J.A.Boatz, S.T.Elbert, M.S.Gordon, J.H.Jensen, S.Koseki, N.Matsunaga, K.A.Nguyen, S.Su, T.L.Windus, M.Dupuis, J.A.Montgomery J. Comput. Chem., 14, 1347-1363(1993).
"Advances in electronic structure theory: GAMESS a decade later" M.S.Gordon, M.W.Schmidt pp. 1167-1189, in "Theory and Applications of Computational Chemistry: the first forty years" C.E.Dykstra, G.Frenking, K.S.Kim, G.E.Scuseria (editors), Elsevier, Amsterdam, 2005.
The latter book chapter can be downloaded in PDF format, from here. Unfortunately, the journal containing the 1993 paper does not yet go back as far as 1993 in its online access, but your library will have this paper in its stacks.
The GAMESS program documentation contains references to the methods and implementations of various GAMESS capabilities, particularly in the fourth chapter, "Further Information". If you use any of the unusual features in GAMESS, it is considerate to quote the individual paper(s) describing those features, in addition to citing the program itself.