AI-generated illustration accompanying text

A very interesting form. First time I saw this was Ryan Moulton’s Tour of the Sacred Library, via Michael Nielsen.

Tour of the Sacred Library

It’s… quite arresting. A key insight here is that Ryan generates these images by referring the model to James Gurney, the illustrator of Dinotopia and author of Imaginative Realism.

More specifically, the images are generated using VQGAN for generation and CLIP for directing the image to match a text string (“<room name> in the sacred library by James Gurney”). Note that the room name doesn’t actually include the full language of the accompanying prose.