The Phantom Slasher did tell Rhue "Were it not for me, you would not even have the capacity to create memories".. So I would have thought that Rhue creates the memories himself using the memories of others.
Even ones own memories becomes fragmented over time, there was bound to be some distortion for Rhue, God knows how long he's been about swapping memories and taken them as truth...