What is Google Deep Dream? (Basic Theory) | Deep Dream Series #1
❤️ Become The AI Epiphany Patreon ❤️ ► https://www.patreon.com/theaiepiphany
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

Learn the basic theory behind the Deep Dream algorithm.

You'll learn about:
✔️ What Google Deep Dream is and how it all began
✔️Impacts of shallower/deeper layers on the final result
✔️Basic concepts: gradient ascent, image pyramids, etc.

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
✅ DeepDream in PyTorch: https://github.com/gordicaleksa/pytorch-deepdream
✅ Original blog: https://ai.googleblog.com/2015/06/inceptionism-going-deeper-into-neural.html
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

⌚️ Timetable: 
1:10 - How it all started in 2015
3:30 - Simple explanation of how it works
4:30 - Impact of using shallow layers
5:30 - Dataset matters (ImageNet vs Places 365)
6:10 - More details on how it works

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
💰 BECOME A PATREON OF THE AI EPIPHANY ❤️

If these videos, GitHub projects, and blogs help you,
consider helping me out by supporting me on Patreon!

The AI Epiphany ► https://www.patreon.com/theaiepiphany
One-time donations: https://www.paypal.com/paypalme/theaiepiphany

Much love! ❤️

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

💡 The AI Epiphany is a channel dedicated to simplifying the field of AI using creative visualizations and in general, a stronger focus on geometrical and visual intuition, rather than the algebraic and numerical "intuition". 

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
👋 CONNECT WITH ME ON SOCIAL 
LinkedIn ► https://www.linkedin.com/in/aleksagordic/ 
Twitter ► https://twitter.com/gordic_aleksa 

Instagram ► https://www.instagram.com/aiepiphany/ 
Facebook ► https://www.facebook.com/aiepiphany/ 

👨‍👩‍👧‍👦 JOIN OUR DISCORD COMMUNITY:
Discord ► https://discord.gg/peBrCpheKE

📢 SUBSCRIBE TO MY MONTHLY AI NEWSLETTER:
Substack ► https://aiepiphany.substack.com/

💻 FOLLOW ME ON GITHUB FOR COOL PROJECTS: 
GitHub ► https://github.com/gordicaleksa 

📚 FOLLOW ME ON MEDIUM:
Medium ► https://gordicaleksa.medium.com/ 
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

#deepdream #deeplearning #art #ai
 I'm starting a new video series with this video about the algorithm called Deep Dream. And in this video we're going to cover the basic theory behind the algorithm. And in the next one we'll go through the code I just open sourced a couple of days ago. As you can see it here on the screen. And it will give you some really psychedelic looking imagery. But that's not the point obviously of this series. It's going to teach you, you're going to learn a lot more about deep learning itself, about like computational graphs in PyTorch and also how to create some really nice artwork using this code. But first let's jump to some basic theory about the Deep Dream algorithm itself. So what's the idea behind this algorithm? So given an input image the thing that happens is that the neural network pretrained on some classification test or something like on ImageNet or MIT places 365. Basically sees something in that image, certain features. And whatever the network sees we just tell it to just amplify those exact features. And by doing that you get the image on the right that you can see on the screen. I found the story how it all started really interesting. So Mordvinsa, the guy that actually created the Deep Dream algorithm, the initial idea, actually woke up like around 2am and had a nightmare. He just sat down on his computer, coded this up in maybe 15 minutes and instantaneously he got some really interesting results. What's even more uncommon maybe is that they did not publish a paper on this, they just wrote the blog and subsequently they open sourced a simple implementation of the Deep Dream algorithm. And you can see on the screen they started getting some really weird creatures and results titled like Pig Snail or the Dogfish. It just sparked a lot of imagination from the very beginning. And those creatures came from this cloud image you can see in the top left. And also the interesting thing is that depending on what you give to the network, what you input as the input, for example if you give it some scene it will start populating it with some towers or buildings and if you give it some kind of maybe leaf or something it will start populating it with insects and animals. So it also depends on the data that the network itself was trained on, not only on the input image. But I'll get into a bit more details a bit later. But Deep Dream was not only about creating these psychedelic crazy looking images, it was actually a part of the effort to better understand neural networks, a field called interpretability of neural networks. And you can see in the bottom left those examples, so what they actually told the network was hey, tell me what do you see, what do you think dumbbell is? And going in reverse and reconstructing the image they got these four images in the bottom left. And you can see that the network actually learned to always expect to see some muscular hands of a weightlifter next to the dumbbell or a bodybuilder, whatever, and did not manage to actually extract the concept itself of a dumbbell, how we humans actually think about the dumbbell itself as an independent object, not dependent on the context where we usually see it. And let's get a bit deeper into how the thing actually works. So what you do, you have a pre-trained neural network as I already said, pre-trained on some classification data set like ImageNet or MIT places 365, and you pass in the input image and you feed it through to a certain layer, and when you come to a certain layer, whatever activations you have there, you want to amplify them. And you do that by doing the gradient ascent and not descent. For example, you took a simple L2 or MSC loss on those activations and you wanted to do a classical procedure that's a minimization. What you would end up with would be that the input image would either become black or more probably just some random noise image. On the other hand, if you tell it to maximize instead of minimize those activations, you get the deep dream results that we're familiar with as the image on the right here. Now the interesting thing is that depending on which exact layer in the network you're trying to optimize, maximize in this case, you get different interesting results. So if you take some shallower layers from some of the first layers in the neural network and you try and do this procedure I just described, you'll get this low level patterns, really interesting patterns and colors, edges and corners, the stuff that we know, the usual low level vision stuff. But on the other hand, if you took some deeper layer in the network, you would start getting some more increasingly complex features such as like maybe animal eyes or snout or something. I'm assuming of course here that the network was trained on ImageNet which contained a lot of like images of dogs and cats and animals in general. But if we had some other data set, then those high level features would be some other things and not eyes or snouts. Now I already mentioned ImageNet and MIT places 365, but those are not the only data sets out there obviously. You can use some other ones, but just that I use these two and these two are really common in the deep dream community. So you can see on the left here, the image was produced by a network that was trained on a lot of imagery that contained animals. On the right you can see the network that was trained on MIT places which contains some scenery, human built objects, pubs, cafes and stuff like that. And that's the reason it has these human like built things ingrained into the image itself. Going a bit more deeper into the theory, so I already mentioned gradient ascent. So basically the only difference is you just change the sign. When you do the update for a single pixel, you don't do the minus where you use the learning rate and the gradients, you just switch it to plus. But there are also some more advanced stuff like image pyramid. So the idea there is, so first what's image pyramid? So it's basically you have your base image and you just down sample it by some ratio, let's say 1.5 and then you do it again and again and again and you just stack those images and that's the image pyramid. So now why do we do that? Now the thing is there is something called receptive field of network and when you feed all of those images from the image pyramid with different sizes to the network, you will see different features in all of them. Some will be more fine grained, some will be like coarser features and combining all of those across the layers, you'll get a really nice looking deep dream image. So why they work is that when you have a smaller image in the input, a single neuron somewhere in some deeper layer will be able to cover most of the pixels in that input image and it will be able to manipulate and tweak those pixels. Whereas when you have bigger image, that pixel will only be able to change and modify just a small area of the input image. So the algorithm proceeds like this. You just start with the smallest resolution, you do the deep dream, then you just up sample the image, then again you do the deep dream and then again until you get to the base image of the pyramid, the biggest resolution and you just stop. There are two more important things that you need to implement in order to have some nice images. One is the gradient smoothing and normalization. So basically if you only took gradients that you get from this optimization and just blindly applied them, the image will quickly explode out of the bounds that we are keeping it in. So what you have to do is to just treat the gradients as a simple image and take a Gaussian kernel and just filter out the image, your convolution with the kernel and that will just smoothen out the gradients and additionally you just want to take the standard deviation of the whole thing and just scale all of those gradients with that standard deviation and that will just keep the things scoped. Aside from that it's quite usual to see to just after applying the gradient ascent step, you just want to clip the image so that you keep it in the designated boundary that you want to keep it in and those are usually ImageNet normalized boundaries. That should be enough for you to understand the next coding video where we'll go step through the code and just try and run this ourselves and get some really awesome looking images. Yeah and by the way don't forget to check out the description where I've linked GitHub code. Feel free to just play around with it before I create the actual video of explaining how to use it. I guess most of you are coders or experienced machine learning engineers already so go ahead and play. And as always if you found this video useful and only if you found it useful please consider subscribing and sharing the video. That would mean a lot to me and yeah see you in the next video.
[{"start": 0.0, "end": 5.92, "text": " I'm starting a new video series with this video about the algorithm called Deep Dream."}, {"start": 5.92, "end": 10.72, "text": " And in this video we're going to cover the basic theory behind the algorithm."}, {"start": 10.72, "end": 14.72, "text": " And in the next one we'll go through the code I just open sourced a couple of days ago."}, {"start": 14.72, "end": 16.8, "text": " As you can see it here on the screen."}, {"start": 16.8, "end": 20.240000000000002, "text": " And it will give you some really psychedelic looking imagery."}, {"start": 20.240000000000002, "end": 24.76, "text": " But that's not the point obviously of this series."}, {"start": 24.76, "end": 29.64, "text": " It's going to teach you, you're going to learn a lot more about deep learning itself, about"}, {"start": 29.64, "end": 36.84, "text": " like computational graphs in PyTorch and also how to create some really nice artwork using"}, {"start": 36.84, "end": 37.84, "text": " this code."}, {"start": 37.84, "end": 41.68, "text": " But first let's jump to some basic theory about the Deep Dream algorithm itself."}, {"start": 41.68, "end": 43.6, "text": " So what's the idea behind this algorithm?"}, {"start": 43.6, "end": 50.72, "text": " So given an input image the thing that happens is that the neural network pretrained on some"}, {"start": 50.72, "end": 57.16, "text": " classification test or something like on ImageNet or MIT places 365."}, {"start": 57.16, "end": 61.0, "text": " Basically sees something in that image, certain features."}, {"start": 61.0, "end": 67.39999999999999, "text": " And whatever the network sees we just tell it to just amplify those exact features."}, {"start": 67.39999999999999, "end": 71.03999999999999, "text": " And by doing that you get the image on the right that you can see on the screen."}, {"start": 71.03999999999999, "end": 73.64, "text": " I found the story how it all started really interesting."}, {"start": 73.64, "end": 80.67999999999999, "text": " So Mordvinsa, the guy that actually created the Deep Dream algorithm, the initial idea,"}, {"start": 80.67999999999999, "end": 83.88, "text": " actually woke up like around 2am and had a nightmare."}, {"start": 83.88, "end": 89.19999999999999, "text": " He just sat down on his computer, coded this up in maybe 15 minutes and instantaneously"}, {"start": 89.19999999999999, "end": 92.08, "text": " he got some really interesting results."}, {"start": 92.08, "end": 96.92, "text": " What's even more uncommon maybe is that they did not publish a paper on this, they just"}, {"start": 96.92, "end": 102.32, "text": " wrote the blog and subsequently they open sourced a simple implementation of the Deep"}, {"start": 102.32, "end": 103.96, "text": " Dream algorithm."}, {"start": 103.96, "end": 108.96, "text": " And you can see on the screen they started getting some really weird creatures and results"}, {"start": 108.96, "end": 113.08, "text": " titled like Pig Snail or the Dogfish."}, {"start": 113.08, "end": 116.28, "text": " It just sparked a lot of imagination from the very beginning."}, {"start": 116.28, "end": 121.44, "text": " And those creatures came from this cloud image you can see in the top left."}, {"start": 121.44, "end": 125.84, "text": " And also the interesting thing is that depending on what you give to the network, what you"}, {"start": 125.84, "end": 131.92, "text": " input as the input, for example if you give it some scene it will start populating it"}, {"start": 131.92, "end": 138.96, "text": " with some towers or buildings and if you give it some kind of maybe leaf or something it"}, {"start": 138.96, "end": 142.18, "text": " will start populating it with insects and animals."}, {"start": 142.18, "end": 148.88, "text": " So it also depends on the data that the network itself was trained on, not only on the input"}, {"start": 148.88, "end": 149.88, "text": " image."}, {"start": 149.88, "end": 151.68, "text": " But I'll get into a bit more details a bit later."}, {"start": 151.68, "end": 157.24, "text": " But Deep Dream was not only about creating these psychedelic crazy looking images, it"}, {"start": 157.24, "end": 163.72, "text": " was actually a part of the effort to better understand neural networks, a field called"}, {"start": 163.72, "end": 166.52, "text": " interpretability of neural networks."}, {"start": 166.52, "end": 173.04000000000002, "text": " And you can see in the bottom left those examples, so what they actually told the network was"}, {"start": 173.04000000000002, "end": 178.96, "text": " hey, tell me what do you see, what do you think dumbbell is?"}, {"start": 178.96, "end": 184.04000000000002, "text": " And going in reverse and reconstructing the image they got these four images in the bottom"}, {"start": 184.04000000000002, "end": 185.04000000000002, "text": " left."}, {"start": 185.04000000000002, "end": 191.60000000000002, "text": " And you can see that the network actually learned to always expect to see some muscular"}, {"start": 191.6, "end": 197.79999999999998, "text": " hands of a weightlifter next to the dumbbell or a bodybuilder, whatever, and did not manage"}, {"start": 197.79999999999998, "end": 202.88, "text": " to actually extract the concept itself of a dumbbell, how we humans actually think about"}, {"start": 202.88, "end": 208.4, "text": " the dumbbell itself as an independent object, not dependent on the context where we usually"}, {"start": 208.4, "end": 209.4, "text": " see it."}, {"start": 209.4, "end": 212.78, "text": " And let's get a bit deeper into how the thing actually works."}, {"start": 212.78, "end": 217.88, "text": " So what you do, you have a pre-trained neural network as I already said, pre-trained on"}, {"start": 217.88, "end": 225.0, "text": " some classification data set like ImageNet or MIT places 365, and you pass in the input"}, {"start": 225.0, "end": 231.07999999999998, "text": " image and you feed it through to a certain layer, and when you come to a certain layer,"}, {"start": 231.07999999999998, "end": 235.2, "text": " whatever activations you have there, you want to amplify them."}, {"start": 235.2, "end": 238.76, "text": " And you do that by doing the gradient ascent and not descent."}, {"start": 238.76, "end": 245.88, "text": " For example, you took a simple L2 or MSC loss on those activations and you wanted to do"}, {"start": 245.88, "end": 249.84, "text": " a classical procedure that's a minimization."}, {"start": 249.84, "end": 255.56, "text": " What you would end up with would be that the input image would either become black or more"}, {"start": 255.56, "end": 258.44, "text": " probably just some random noise image."}, {"start": 258.44, "end": 264.2, "text": " On the other hand, if you tell it to maximize instead of minimize those activations, you"}, {"start": 264.2, "end": 269.52, "text": " get the deep dream results that we're familiar with as the image on the right here."}, {"start": 269.52, "end": 274.84, "text": " Now the interesting thing is that depending on which exact layer in the network you're"}, {"start": 274.84, "end": 281.08, "text": " trying to optimize, maximize in this case, you get different interesting results."}, {"start": 281.08, "end": 287.23999999999995, "text": " So if you take some shallower layers from some of the first layers in the neural network"}, {"start": 287.23999999999995, "end": 294.35999999999996, "text": " and you try and do this procedure I just described, you'll get this low level patterns, really"}, {"start": 294.35999999999996, "end": 299.03999999999996, "text": " interesting patterns and colors, edges and corners, the stuff that we know, the usual"}, {"start": 299.03999999999996, "end": 302.88, "text": " low level vision stuff."}, {"start": 302.88, "end": 308.68, "text": " But on the other hand, if you took some deeper layer in the network, you would start getting"}, {"start": 308.68, "end": 316.56, "text": " some more increasingly complex features such as like maybe animal eyes or snout or something."}, {"start": 316.56, "end": 320.71999999999997, "text": " I'm assuming of course here that the network was trained on ImageNet which contained a"}, {"start": 320.71999999999997, "end": 325.28, "text": " lot of like images of dogs and cats and animals in general."}, {"start": 325.28, "end": 331.15999999999997, "text": " But if we had some other data set, then those high level features would be some other things"}, {"start": 331.16, "end": 332.94, "text": " and not eyes or snouts."}, {"start": 332.94, "end": 338.84000000000003, "text": " Now I already mentioned ImageNet and MIT places 365, but those are not the only data sets"}, {"start": 338.84000000000003, "end": 340.40000000000003, "text": " out there obviously."}, {"start": 340.40000000000003, "end": 345.40000000000003, "text": " You can use some other ones, but just that I use these two and these two are really common"}, {"start": 345.40000000000003, "end": 347.56, "text": " in the deep dream community."}, {"start": 347.56, "end": 352.48, "text": " So you can see on the left here, the image was produced by a network that was trained"}, {"start": 352.48, "end": 355.72, "text": " on a lot of imagery that contained animals."}, {"start": 355.72, "end": 360.72, "text": " On the right you can see the network that was trained on MIT places which contains some"}, {"start": 360.72, "end": 365.52000000000004, "text": " scenery, human built objects, pubs, cafes and stuff like that."}, {"start": 365.52000000000004, "end": 372.24, "text": " And that's the reason it has these human like built things ingrained into the image itself."}, {"start": 372.24, "end": 378.48, "text": " Going a bit more deeper into the theory, so I already mentioned gradient ascent."}, {"start": 378.48, "end": 383.02000000000004, "text": " So basically the only difference is you just change the sign."}, {"start": 383.02000000000004, "end": 387.88000000000005, "text": " When you do the update for a single pixel, you don't do the minus where you use the learning"}, {"start": 387.88, "end": 392.8, "text": " rate and the gradients, you just switch it to plus."}, {"start": 392.8, "end": 396.71999999999997, "text": " But there are also some more advanced stuff like image pyramid."}, {"start": 396.71999999999997, "end": 400.64, "text": " So the idea there is, so first what's image pyramid?"}, {"start": 400.64, "end": 406.84, "text": " So it's basically you have your base image and you just down sample it by some ratio,"}, {"start": 406.84, "end": 412.6, "text": " let's say 1.5 and then you do it again and again and again and you just stack those images"}, {"start": 412.6, "end": 414.04, "text": " and that's the image pyramid."}, {"start": 414.04, "end": 415.84, "text": " So now why do we do that?"}, {"start": 415.84, "end": 421.15999999999997, "text": " Now the thing is there is something called receptive field of network and when you feed"}, {"start": 421.15999999999997, "end": 426.23999999999995, "text": " all of those images from the image pyramid with different sizes to the network, you will"}, {"start": 426.23999999999995, "end": 429.2, "text": " see different features in all of them."}, {"start": 429.2, "end": 434.59999999999997, "text": " Some will be more fine grained, some will be like coarser features and combining all of"}, {"start": 434.59999999999997, "end": 439.41999999999996, "text": " those across the layers, you'll get a really nice looking deep dream image."}, {"start": 439.41999999999996, "end": 445.23999999999995, "text": " So why they work is that when you have a smaller image in the input, a single neuron somewhere"}, {"start": 445.24, "end": 451.56, "text": " in some deeper layer will be able to cover most of the pixels in that input image and"}, {"start": 451.56, "end": 456.08, "text": " it will be able to manipulate and tweak those pixels."}, {"start": 456.08, "end": 462.24, "text": " Whereas when you have bigger image, that pixel will only be able to change and modify just"}, {"start": 462.24, "end": 464.52, "text": " a small area of the input image."}, {"start": 464.52, "end": 466.46000000000004, "text": " So the algorithm proceeds like this."}, {"start": 466.46000000000004, "end": 471.06, "text": " You just start with the smallest resolution, you do the deep dream, then you just up sample"}, {"start": 471.06, "end": 476.04, "text": " the image, then again you do the deep dream and then again until you get to the base image"}, {"start": 476.04, "end": 480.0, "text": " of the pyramid, the biggest resolution and you just stop."}, {"start": 480.0, "end": 484.72, "text": " There are two more important things that you need to implement in order to have some nice"}, {"start": 484.72, "end": 485.72, "text": " images."}, {"start": 485.72, "end": 489.24, "text": " One is the gradient smoothing and normalization."}, {"start": 489.24, "end": 495.56, "text": " So basically if you only took gradients that you get from this optimization and just blindly"}, {"start": 495.56, "end": 501.88, "text": " applied them, the image will quickly explode out of the bounds that we are keeping it in."}, {"start": 501.88, "end": 509.88, "text": " So what you have to do is to just treat the gradients as a simple image and take a Gaussian"}, {"start": 509.88, "end": 516.84, "text": " kernel and just filter out the image, your convolution with the kernel and that will"}, {"start": 516.84, "end": 522.64, "text": " just smoothen out the gradients and additionally you just want to take the standard deviation"}, {"start": 522.64, "end": 530.16, "text": " of the whole thing and just scale all of those gradients with that standard deviation and"}, {"start": 530.16, "end": 532.96, "text": " that will just keep the things scoped."}, {"start": 532.96, "end": 539.6, "text": " Aside from that it's quite usual to see to just after applying the gradient ascent step,"}, {"start": 539.6, "end": 544.4399999999999, "text": " you just want to clip the image so that you keep it in the designated boundary that you"}, {"start": 544.4399999999999, "end": 548.84, "text": " want to keep it in and those are usually ImageNet normalized boundaries."}, {"start": 548.84, "end": 554.24, "text": " That should be enough for you to understand the next coding video where we'll go step"}, {"start": 554.24, "end": 559.48, "text": " through the code and just try and run this ourselves and get some really awesome looking"}, {"start": 559.48, "end": 560.48, "text": " images."}, {"start": 560.48, "end": 566.2, "text": " Yeah and by the way don't forget to check out the description where I've linked GitHub"}, {"start": 566.2, "end": 567.9200000000001, "text": " code."}, {"start": 567.9200000000001, "end": 573.0400000000001, "text": " Feel free to just play around with it before I create the actual video of explaining how"}, {"start": 573.0400000000001, "end": 574.0400000000001, "text": " to use it."}, {"start": 574.0400000000001, "end": 578.8000000000001, "text": " I guess most of you are coders or experienced machine learning engineers already so go ahead"}, {"start": 578.8, "end": 579.8, "text": " and play."}, {"start": 579.8, "end": 585.28, "text": " And as always if you found this video useful and only if you found it useful please consider"}, {"start": 585.28, "end": 587.76, "text": " subscribing and sharing the video."}, {"start": 587.76, "end": 612.56, "text": " That would mean a lot to me and yeah see you in the next video."}]