Editor’s note: This article is from WeChat public account “Qubit” (ID: QbitAI), author Guo Yi 璞, 36 氪 released with permission.Looking at pictures is always easier than reading words.For example, it is easier and easier to read a comic adapted from a novel than to read a text novel.So, how to turn a story into a comic automatically?AI can already do it.Give it a story, it can tell the story with pictures, with a little modification, it becomes a set of comics.A story like this: A long time ago, a little boy lost his way in the forest. He walked for a long time and was all hungry. He wanted to eat the cake made by his mother, but this time the little boy could n’t go back, so heI drew a large loaf of bread on the ground and ordered sesame seeds on the loaf.The little boy looked at it, as if he had eaten a flat cake, so he didn’t feel that he was very hungry, and stood up again and walked along a small path …Soon, the little boy’s parents found him, and they came home together and ate the really delicious cakes.AI can turn it into a comic: In addition, this form of comic can also serve as a storyboard in the film industry, assisting filmmakers in artistic creation.Finding a storytelling diagram So how does this process work?First of all, it should be explained that these pictures are not drawn by AI, but rather a simpler and more convenient way: find out some similar ones from the existing gallery and change them.The gallery here is called the GraphMovie dataset, and the data source is some movie review websites.But there are many pictures in the data set. How can you use AI to automatically find the picture that best matches your story?A model is used here, called a context-aware dense matching model (CADM).It looks like this: According to the content of the story, CADM found some images like this: In addition, there is a model called No Context, which is the best performing AI in the field of “looking at characters and looking for pictures”.No Context also found some images: However, the pictures are still a little bit. Others are also a complete story. You can’t always take such a few pictures to finish, so the story has no details and the reader’s experience is not good.Now, please come up with a third model: the greedy decoding algorithm, which is responsible for finding the pictures that can be used according to the details in the story.After making it come and supplement it, the story is complete immediately: the style of the picture is unified, but with such an effect, you will be confused, as if it cannot convey the story in the previous text.Where is the problem?The first problem is that there are many background, environment and other related elements in the picture, which have nothing to do with the main line of the story. Seeing it will also affect your understanding of the story.These redundant elements need to be deleted. Here, He Kaiming’s famous work Mask R-CNN is used to segment the area and delete the parts of the picture that are not related to the story.Now, these pictures look like this: The second problem is that the style of these pictures is too different. Taking out such comics will be hit by readers.Therefore, it is necessary to unify the style of the pictures. Here, a tool CartoonGAN is used, which can be understood literally. This is a GAN (Generating Adversarial Network) that makes pictures into cartoon style.After the cartoon GAN processing, this set of pictures became like this: it seems better, but there is a big bug: these people look different!You say they are the protagonists of the same story, and I don’t believe it.So, the third question comes, how can we make these people look the same?Here, the researchers directly found a software called Autodesk Maya, which is a software for processing 3D images in movies. It relies on it to create 3D scenes, characters, and props. It uses a semi-manual method to put people in 9 pictures.It’s all changed.However, the author of the paper said that the process of making 3D images in the future is expected to be fully automatic.This meal operation is really fierce, with nine pictures that can’t hit the edge. Now the style is consistent, the story is smooth, and even the background and landscaping are added.There are great uses in the film industry. In fact, the “comic” generated in this way is not the end result.It’s actually used to make movies.During the preparation of the movie, a demo called “storyboard” is required.With the help of storyboards, filmmakers can change the demo during the creative process, finish the finished product after setting it, and put the tearing process in front to prevent the father of Party A from submitting amendments after the completion, resulting in a sharp increase in workload..Therefore, as in this paper, automatically generating storyboards can save filmmakers a lot of time and increase the productivity of creators.The team of authors of this paper produced by Dr. NPC is very large. There are 9 authors from Renmin University of China, Microsoft and Beijing Film Academy.The first work, Chen Shizhe, is now in his fifth year of doctoral degree at the National People’s Congress. He has also interned with the Microsoft Xiaobing team and has also visited CMU and the University of Adelaide.She is also an academic expert, and this year alone, she has published three top-notch writings including this one.In addition, Song Ruihua, the chief scientist of Microsoft’s Xiaobing team, also participated in this research.Dr. Song Ruihua graduated from Tsinghua University with a long-term research in the fields of short text dialogue and generation, information retrieval and extraction. He has served as the program chair or senior program chair of SIGIR, SIGKDD, CIKM, WWW, WSDM and other conferences.Portal Neural Storyboard Artist: Visualizing Stories with Coherent Image Sequences Author: Shizhe Chen, Bei Liu, Jianlong Fu, Ruihua Song, Qin Jin, Pingping Lin, Xiaoyu Qi, Chunting Wang, Jin Zhouhttps: //arxiv.org/abs/1911.10460v1.