Text to Art: Using NLP to Create Art

10/03/2022
blog

From finance and marketing to higher education, AI has taken over the entire world. The world of creators and artists is no different. If someone told you a few years ago that you can convert text to art, you would probably call them mental and laugh off the idea. Today, however, it is a reality that has opened doors for tons of benefits for the artist community.

The idea has become immensely popular due to the appearance of NFTs in the market. NFTs have started an era of digital artwork, allowing the artists to tokenize and sell their work to a quickly growing marketplace. For instance, Art AI has recently introduced Eponym, a software that allows text to art conversion, and direct creation of NFTs.

Since digital artwork is the new tomorrow, we have decided to explore the concepts of converting text to art by using NLP (Natural Language Processing). So, let’s begin, shall we?

What is Natural Language Processing (NLP)?

Natural Language Processing is a branch of Artificial Intelligence (AI) that allows computers and machines to understand texts and spoken words. NLP allows machines to understand the actual meaning of the data, including sarcasm, sentiment, and intent.

With NLP, humans can communicate with the computers in their intrinsic language, and by using statistical, machine learning, and deep learning models, NLP interprets the command and produces the results.

While the concept may sound too complex, you will be surprised to find out that you have encountered NLP several times in your life, in fact, in your day today. Did you use voice-operated GPS while going to work today? Or did you need Google Translate to find out what your colleague was telling you in his native language?  From virtual agents, like Siri and Alexa, to customer service chat boxes that pop up when you visit your favourite websites, NLP is everywhere!

Why is NLP being used to Convert Text to Art?

Artists and creators are known to be unique personalities that have god-gifted methods of expressing themselves. On the other hand, AI is a highly logical concept with no space for emotion whatsoever. Combining the both might seem illogical, yet it is a concept being widely accepted due to the advent of NFTs.

Don’t know about NFTs? Visit Vidalgo, where we talk about NFTs, their market growth, trends in this field and much more!

Click on the animation for more options

NFTs have converted artwork into a source of money. And like any other money-generating source, people are racing to double it. While traditional artwork might require days to create, an NLP-based artwork can take minutes, if not seconds. It has also opened doors for non-artists to enjoy the OpenSea marketplace. You no longer need to be a professional to earn from NFTs; you just have to have a thought and voila! –you have your art.

But What about Creativity?

Many people see the use of AI in creation as a threat to the creators. While the artist community is still an underdeveloped and under-appreciated part of the world, there should be no doubt about the fact that, like any other community, they need to keep up with the world.

According to Forbes, “it is becoming harder and harder to deny that artificial intelligence is capable of creativity.” Yet there is no doubt that this creativity stems from the creativity of a human being himself.

NLP-based artwork can never take away an artist’s job because an artist is an initiator for it. His creativity is the only thing such software can display. Without his vision, NLP-based art is nothing. Furthermore, a human is needed to remove racism, hate or inappropriate elements from the artwork that logic-based software may generate.

DALL-E –A NLP Project by OpenAI

DALL-E is an extension of GPT-3, which was a language model that created human-like text. The results of GPT-3 make text written by humans and machines indistinguishable. Working in the same line, OpenAI introduced DALL-E, which can not only generate an original image but can also extend an image towards the bottom if commanded. Just like GPT-3, DALL-E is a transformer. A transformer uses the attention mechanism, giving different levels of importance to different parts of inputs.

The word DALL-E is the scientists’ way of giving tribute to the Spanish artist Salvador Dali and Pixar’s beloved 2008 robot, WALL-E.  From the looks of it, DALL-E can interpret commands that are far from reality, for instance, “an arm-chair in the shape of an avocado.”

Text to Art: avocado-shaped arm chair
Image Courtesy: Open AI

The fact that this AI can combine entirely unrelated textual commands and convert them into an art piece can make anyone dumb-founded, and that’s what scientists at OpenAI feel like. Since the field is quite unexplored, their creation has several limitations. The software creates a different image or piece of digital art every time the user rephrases the same command. However, the breakthrough is big enough for anyone to ponder too much on the limitations for now.

How does DALL-E convert text to art?

DALL-E requires user input in the form of text and an image input from a program called CLIP. DALL-E divides the input into tokens. For a human being, each letter of the alphabet is a token. For DALL-E, there are 256 BPE-encoded tokens and 1024 image tokens. When a stream of input goes into the program, it divides it into a maximum of 1028 tokens. It then tries to generate the details present in the maximum number of tokens individually.

In this way, the program tries its best to create the maximum number of details mentioned in the input caption. However, it does not always interpret the text correctly; for instance, it cannot create shapes such as a pentagon. Also, it has trouble creating a larger object sitting on a smaller one or correctly understanding the words “standing left off”, “standing below”, and of the likes.

Similarly, the program can create word strings. However, the longer the string, the lesser will be its accuracy. The program also has trouble distinguishing between colours, but don’t we all?

The Potential of NLP and AI Artwork on the Future

With DALL-E and its successors, artists of every kind will be able to fare better. They can save their time and supplies. Corrections to their artworks will be easier, and brainstorming will be a piece of cake.

Fashion designers no longer need to rely on digital or pencil sketches, and they can just tell the program how many flares the skirt should have. Logo designers no longer need to put an effort to place their logos on several cards and shop fronts to check how they look, and they can tell the program to create their design with multiple backgrounds.  

Similarly, game designers can create their otherworld characters and see exactly what kind of beak fits in the place of a nose. The possibilities are endless! So, are you ready to explore them?

Visit our blog for more information about machine learning and deep learning in animation and artwork.

Other Posts

blog
Best Lottie Players, Plugins and Tools
30/04/2022
Lottie has revolutionized the world of animation. To give a very basic definition, Lottie is a file format specific for animations, much like GIF. Due to Lottie animations, users can now display high-quality animations without any problems, such as pixelation, large file size, or slow webpage loading speeds. A single Lottie file is many times […]
blog
Infographics Trends and Animated Infographics in 2022
23/04/2022
With the rise of social media usage, visual content is becoming more and more popular than mere texts. From informational videos to sending memes to your friends and family, visual content is what people love nowadays. A popular type of visual content is infographics- packed with information, made to be attractive to the audience. Want […]
blog
7 Best Lottie Marketplaces To Help With Your Projects
16/04/2022
The Internet has brought the world closer, and sharing information has become extremely easy in this day and age. Today, there are numerous methods that let you share information, from videos to text and infographics to images. One such method is the use of animations which gives lets you provide information in a highly expressive […]

Deep Learning Animation

25/11/2021
blog

What is deep learning?

Deep learning animation is changing the way animation is done in the world. If you’re new to the idea, it may sound like the ominous lead up to destruction in the latest Hollywood flick. But don’t worry! We’ll unpack what it means and share an optimistic take on how AI animation can benefit any artist.

So what is deep learning, you ask? According to IBM, deep learning is a subset of machine learning. What’s that? Machine learning is a man-made neural network with three or more layers. These neural networks attempt to simulate the behavior of the human brain, allowing it to “learn” from large amounts of data.

Think about self-driving cars- they have the ability to learn new routes, traffic patterns, and obstacles and they use that information to get you to your destination safely. And that’s a lot of information! Deep learning is increasing how much complexity an AI can process at once. When it comes to deep learning animation, we take it a step further.

It even creates the capacity for an animation program to learn an animator’s style, design interests, and visual concept and create something new from it.

What does it look like for animation?

Let’s get concrete about the difference deep learning animation can make. In the early 1990’s the average animated film took several years to complete. Pixar’s Toy Story in particular took 7 years to create. And that was using the most sophisticated technology of the day. For contrast, the 2019 movie Toy Story 4, was on pace to take 2 years, before the crew discarded about ¾ to rework the story.

Still curious? Check out this video about the potential for deep learning

Deep learning blends different kinds of machine learning. By layering different kinds of learning, we are growing the capacity to create tools for artists.

How deep does deep learning go?

Deep learning technology lies behind everyday products.  Chances are you run into deep learning technology on the daily without even knowing it. It’s behind most of the services we’ve become accustomed to. Most people can’t afford an assistant, but we can all appreciate that Siri or Alexa wakes us up in the morning, reminds us of our appointments, and even puts together our shopping lists. 

Truly everything from digital assistants, voice enabled devices, and even simple card readers are powered by this technology.  Deep learning animation is just the next step. This time to support the creatives of the world. So, yes, AI has become integral to how we work and live.

What is deep learning animation?

Now, can you imagine the concept of a streamlined automation process for AI animation that creates a basic skeleton for a human or four-legged creature? Artificial intelligence opens up new opportunities for animators by being able to present suggested designs for other characters using deep learning. Often, an animator’s most challenging job is to create new characters.

This is where new advancements in AI animation come in. If an existing set of characters (think about Shrek or Monsters Inc) are created, the software can come up with new samples by learning from them. Additionally, deep learning and artificial intelligence can be helpful when it comes to animating the character. The same way it learned about the shape of a face, body, or nose from thousands of characters, deep learning animation can also learn about movement – from dancing to combat.

The coolest part about AI animation?

Imagine having your absolute favorite artist working with you on an animation- guiding your style to be reminiscent of their own, while saving you hours and hours of time trying to replicate their work by hand. That’s the power of style transfer in AI animation.

Yes! You could even replicate that renaissance artist you’re obsessed with. Deep learning animation ultimately opens the door to more possibility for artists.

Style transfer is an optimization technique that can be used to transfer drawing to AI animation instead of by hand. It can let animators transform what they’re creating in the style of someone whose work they admire, like Van Gogh. Style transfer blends a style from one image into another image keeping its content intact.

The only change is the style configurations of the image to give a different artistic touch to your image. The deep learning animation shifts the layout or the sketch and style, be it the style of painting or the specific colors.  No need for the animator to learn every stroke of the style.

Is deep learning animation still real creativity?

Absolutely! When we look at artificial intelligence, it’s natural to wonder about the human element of creativity. When animators started to pick a computer mouse over their pencils, many worried that machines were taking over. But the results have been quite the opposite. Since then, not only do animators have more control over what they produce, but the quality of visual effects and animation has improved dramatically.

Did I mention how much AI animation can do?

An essential part of bringing 3D animated characters to life is the ability to translate their physical movement seamlessly into any scene or environment. AI animation has enhanced artists’ ability to create! Animating characters to naturally interact with objects and the environment requires putting together different types of movements in a complex manner.

And yes, I mean really complex. Motions can greatly differ not only in their postures, but also in their duration, contact patterns, and possible transitions. Most machine learning-based methods for user-friendly character motion have been limited to more simplistic actions.

These simple commands look like commanding an animated character to move from one point to the next. Deep learning animation increases the frequency and complexity of these actions and motions. This AI animation creates seemingly endless options for animators to create more in less time.

The future of deep learning animation

Simply put, we could be closing in on a future where we sit down in a theater and watch a movie that was conceptualized and created by artificial intelligence, and animated as well as rendered by deep learning algorithms. Is that such a bad thing, though?

The world is asking if algorithms, AI animation and robots will start dominating the film industry like they have other industries. While it is true that there are certain tasks that no longer need to be performed manually, the result is a need for even more creative people!

Talented people who can train deep learning algorithms to perform beautifully, and who use AI animation to further enhance their own unique style of art so that the world can enjoy it for everything it deserves to be.

Other Posts

blog
Best Lottie Players, Plugins and Tools
30/04/2022
Lottie has revolutionized the world of animation. To give a very basic definition, Lottie is a file format specific for animations, much like GIF. Due to Lottie animations, users can now display high-quality animations without any problems, such as pixelation, large file size, or slow webpage loading speeds. A single Lottie file is many times […]
blog
Infographics Trends and Animated Infographics in 2022
23/04/2022
With the rise of social media usage, visual content is becoming more and more popular than mere texts. From informational videos to sending memes to your friends and family, visual content is what people love nowadays. A popular type of visual content is infographics- packed with information, made to be attractive to the audience. Want […]
blog
7 Best Lottie Marketplaces To Help With Your Projects
16/04/2022
The Internet has brought the world closer, and sharing information has become extremely easy in this day and age. Today, there are numerous methods that let you share information, from videos to text and infographics to images. One such method is the use of animations which gives lets you provide information in a highly expressive […]

Will artificial intelligence (AI) take over the world of animation?

23/11/2021
blog

Modern technology has become an irreplaceable pillar in our society. With phones, tablets, and laptops always within an arm’s reach, it would be silly to not take advantage of such helpful tools. Technology is used in every aspect of modern life but the one field it has yet to take over is the arts. AI animation has become a popular stepping stone into the field due to the apprehension amongst other art forms. 

As technology advances more and more by the minute, we should not forget that with advancement also comes the potential to uplift and create. Technology is a tool, not a threat, so as long as we embrace the advancements, I believe we can use this technology as a shortcut to lead to more opportunity for artists and animators. 

That leads us to the question; Will Artificial Intelligence take over animation? As a career? No. As a tool? Absolutely. Let’s talk about it.

What is AI?

What comes to mind when I say, “artificial intelligence”?  You might be thinking of sophisticated robots or those text generated scripts that circulate twitter from time to time. And while those are certainly forms of AI, in reality this technology is much more present in our everyday lives. Every time you perform a simple google search, you’re interacting with AI. 

How about speech recognizing software such as Alexa or Siri? You’re having a conversation with AI. Every time you scroll through recommendations on Netflix, you’re looking through a custom panel created for you by AI animation. Even when you’re playing the ‘computer’ in a quick round of chess to kill time, you’re in a simulated game experience run by AI. It can literally be a part of nearly every moment.

So what is “AI” really?

AI or artificial intelligence is any machine that can demonstrate “cognitive” function that mimics the natural intelligence displayed by animals – humans included.

 Although it sounds like a product from science fiction fanfare of the 1950s, the idea of artificial intelligence has been around since ancient times. Ancient Greek mythology tells the story of Talos, a giant bronze human-like figure who protected Crete from possible invaders.  In 1818, Mary Shelley wrote a novel conceiving of creating life and intelligence through electricity, Frankenstein

The ethics of the topic have also been up for debate for just as long. The most popular argument has been that of an artificial overthrow. Where AI could potentially become so powerful that humans would not be able to stop it from achieving its goals. A scary enough thought to become the plot of dozens of hollywood horror films. It’s even created “Man vs Machine” as its own conflict category in literature. 

And our fears have also found themselves in our real world.  Many debate today the ethics of having AI in our homes, helping us perform basic tasks, and helping surveillance.

Artificial Intelligence in Art

Despite these ethical concerns, playing a fictional opponent is a good way to practice your end game. Similarly, asking Siri to set a timer serves as the perfect assistant. So why are people still so uncomfortable with the idea of AI in fields such as the arts? 

Fear of the unknown. From an early age, humans are taught to fear what’s different from us which is why we grew up learning phrases like “stranger danger.” When something as heartfelt and soul-filled as art gets threatened by cold soulless technology, it causes people to panic. Will robots replace artists? Will the already shrinking art fields become more sparse? If not for the artists, who will record the human condition? It can be a scary consideration for those that don’t trust that humans can work with the technology. 

Here’s the catch; fear is the roadblock of progress. Instead of seeing AI art as competition we need to see it as a teammate. AI animation refers to any work assisted by AI. Instead of being seen as a creation it’s more of a collaboration between man and machine. When speaking specifically about art created by AI, even that can’t be seen as original work.

The computers most often rely on machine learning, a specific type of AI that involves feeding computers countless examples of something until they learn the repeating patterns which results in them being able to create their own similar result. Hence, mimicked work.

Instead of seeing AI art as competition we need to see it as a teammate.

Avidor, Vidalgo

AI Animation brings a huge value to artists

To be truly original, one has to create, led by their own experiences. Without a heart behind it, I don’t believe that art created by AI can be considered true art. Art created by AI, although interesting in theory, relies too heavily on examples to be original.

It’s important to use this technology to our advantage. When it comes to AI animation,  it would be silly not to acknowledge the strengths and use them to streamline tedious, repetitive work loads. Most animation companies and studios have been using this technology and developing their own over the past decade.

Specific AI art software can be created to animate a character’s hair to such success that an audience can see individual realistic strands. A feat that would not be possible with production schedules being as strict as they are if it wasn’t for the help of AI animation.

Just take our own tools at Vidalgo.  By no means does Vidalgo take the creator out of the equation.  We simply give the creator the support of AI so they can scale effectively and focus most of their energy on the creation itself.

Final Thoughts

One thing certain about today’s climate is the uncertainty of tomorrow. It’s scary to watch the rapid acceleration of technology without knowing where it’s future is going. If we’ve learned anything about technological advancement, it’s that it can lead to massive job loss.

That leads us back to the main question at hand; Will AI take over animation? 

In my own opinion as a humble artist, I don’t believe it will. It’s an age old question, “Will our creations come to replace us?” I don’t think so. To be human is to live and create. All art comes from a place of personal experience or inspiration. A machine, no matter how hard it tries, can only mimic the original art we give it to study. Yes, art can be patterninistic and repetitive but again, something human inspired those patterns to be where they’re placed. 

There is nothing in the world that can replace the voice of a person with real emotions and lived experience. Each artist is unique and technology won’t be replacing that, only improving it.

Other Posts

blog
Best Lottie Players, Plugins and Tools
30/04/2022
Lottie has revolutionized the world of animation. To give a very basic definition, Lottie is a file format specific for animations, much like GIF. Due to Lottie animations, users can now display high-quality animations without any problems, such as pixelation, large file size, or slow webpage loading speeds. A single Lottie file is many times […]
blog
Infographics Trends and Animated Infographics in 2022
23/04/2022
With the rise of social media usage, visual content is becoming more and more popular than mere texts. From informational videos to sending memes to your friends and family, visual content is what people love nowadays. A popular type of visual content is infographics- packed with information, made to be attractive to the audience. Want […]
blog
7 Best Lottie Marketplaces To Help With Your Projects
16/04/2022
The Internet has brought the world closer, and sharing information has become extremely easy in this day and age. Today, there are numerous methods that let you share information, from videos to text and infographics to images. One such method is the use of animations which gives lets you provide information in a highly expressive […]