That is because, when Allah alone was prayed to (i.e. invoked), you did not believe, and if partners were associated with Him you believed, so judgment belongs to Allah, the High, the Great.