Lore Machine is an AI system built to transform our civilization’s text into multimedia experiences. No forlorn film script in the studio’s slush pile, book in the publisher’s back catalogue, or tale told on your favorite podcast will be left un-visualized!
[Shakes fist at sky]
Lore Machine’s machine requires a lot of software tinkering though. We perform test runs by feeding our pipeline bite-sized literary morsels like short stories, poems, microfiction, nanofiction, postcard fiction and measuring output. Then repeat.
This part of the build is fun, but it’s not like FUN fun. So sometimes we skip ahead and just create a proof-of-concept that articulates exactly how we see stories in our human heads. It could be an anecdote a friend told us converted into an RPG game, song lyrics visualized as an animated music video, or a short story re-imagined as a graphic novel. This is one of those.
Here’s the backstory:
In 1953, legendary science fiction writer Clifford D. Simak published Strangers In The Universe, a short story anthology of galactic depth.
Nestled into that collection lies a shimmering 8-page wonder. Contraption is about an orphan who gets taken in by a couple. It’s complicated though. While his new home should help him feel safe and secure, the couple torment the boy. So now he feels alone AND unloved. Then an alien object lands in his backyard... You can read it for yourself here:
To convert Contraption, the text, into Contraption, the graphic novel, we took 7 basic steps:
Text Synthesis - The distilling of the foundational text. Think of this like a futuristic version of Cole’s notes using all types of Natural Language Processing bells and whistles we are developing at Lore machine.
Character training - The formation of unique and consistent visual character traits, also including some of our new tools!
Wardrobe - ‘Fits gotta be on-point!
Stylistic Configuration - Deep research and application of a style (or combination of styles) to achieve a sustained visual vernacular. Currently this is an entirely human process. But we do gain insights from performing sentiment analysis on foundational texts.
Prompt Generation - The above attributes then rally around the plot points of the narrative, unfurling into a visual storyboard.
Image Generation - A multi-generator approach to producing quality and consistent imagery.
Integration - Voltroning the disparate graphical and textual assets into a format we recognize today. We like graphic novels because they really don’t allow cheating when it comes to sequential dynamics. Aka if there’s a plot hole, you know it.
Right now, this process is arduous. We run into snags and it takes a long time to make it through a production. But we’re learning from the challenges we encounter, and turning those into tools to solve these roadblocks.
We track all these snags in a Text File titled “Things That Are Hard”. Here are three challenges we encountered while producing Contraption:
Fight Scenes / Sex Scenes: Getting consistent characters to interact with each other in the same single-frame graphic output without using manual software (like photoshop) is nearly impossible.
MidJourney Seeds: In a conversation about MidJourney Seeds the other day in the AI Comic Book Creators Facebook page, someone said “Seeds were never really real".” And it’s true. Really the only way to accomplish consistent seeding is to get away from MidJourney and into 1111. This points to a future in which fine control is achieved by combining real-life subjects with AI rotoscoping tools, like those being developed at Runway ML.
Stylistic Bias: To achieve continuity, you gotta stick with one style throughout the story. It’s wobbly at times, as evidenced by Contraption. But you wrestle with it. The trouble is certain styles can have built-in predilection. For instance, wood engraving as a style is very good at depicting nature thanks to its usage in Japanese wood block prints, but insists on casting a despairing shadow over human portraits thanks to its use by genius rainclouds like Fritz Eichenberg.
While there’s a lot of trial and error putting these generative storyscapes together, the joy of seeing stories come alive makes it all very worth it.
At Lore Machine, we’re on a mission to make this text-to-multimedia conversion process easier and faster for storytellers of all types. We believe that if we can help folks bring text closer to visuals - or imagination closer to reality - there can be more stories in the world in myriad accessible formats. Not to emote too hard here, but we also believe that more stories means more sharing of our unique human experiences more fully and maybe more happiness, mystery, empathy and all that good stuff.
If you’ve got a story for the Lore Machine, an idea for a form factor, a tool that might improve our system, or just wanna riff on where this is all going, don’t hesitate to reach out at sup@loremachine.ai.