Introduction to Amazon Lookout for Vision
Hey, welcome to Amazon Lookout for Vision, a new anomaly detection service that helps customers spot product defects using computer vision to automate quality inspection. My name is Anant Patel, I'm the product manager for Lookout for Vision. And I just wanted to paint, start by painting the picture of our a AIML stack. Starting at the bottom we have our frameworks where we enable data scientists to create models in MXNet and PyTorch, we have our middle stack around Sagemaker and Sagemaker studio and GroundTruth that helps to develop models. And at the top we have our AIML services that are fully managed. And one of the call outs that I want to talk about is Amazon Lookout for Vision today, but also to bring notice that we have a broader industrial AI portfolio, Amazon Lookout for equipment, Amazon Lookout for metrics, Monitron, and AWS Panorama. So Lookout for Vision is one of many new industrial AI services, but today's focus will really be around Amazon Lookout for Vision and how we can help customers in this space.
Understanding Customer Challenges
And really the agenda is first to talk about what are some of the challenges that customers are going through, what are they trying to accomplish today? What are the implications of quality assurance and the costs associated with that? Second, we'll go through the service, some of the value proposition and key features that we think will help customers solve some of their problems. We'll talk about common use cases and how customers today are using Amazon Lookout for Vision. We'll go through a demo, well you'll see where we can actually upload images, train a model, and deploy and production in just a few short clicks. And then lastly, we'll talk about some getting started and helpful resources to help you guys get started on your Amazon Lookout for Vision journey. Followed up with some Q and A. So thinking about customers and what is the overall production process today. So for example, you see a car manufacturer on the screen. They have multiple different production points from stamping to welding to vehicle and engine assembly to end of line inspection, and ultimately shipment. Every single component within this flow requires some form of quality assurance. And what we're hearing from customers is that each individual inspection point has its own unique challenges. Sometimes it's easy to do, sometimes it's hard to really find anomalies. And those challenges have real cost implications. So what we are trying to identify is how do we build a service that allows a customer with so many diverse inspection points to utilize a technology where they can insert themselves at these different processes and be able to find defects, automate in an automated fashion, be able to improve their operational efficiency and reduce their operational costs. And really the challenge of doing this is what are the costs of a defective product going down and reaching an end consumer? What are the metrics that they need to actually be mindful of in thinking about how do they better control or reduce the cost per production unit? And the things that we're hearing from customers is really they want to reduce the amount of rework, the scrap, the customer returns and warranty claims. They know that if they're able to control all of those costs in the beginning of the production process, it's gonna save a lot of money, time, and effort on the back end. And so our goal at Lookout for Vision is enabling customers to better control, manage, and ultimately help them decrease costs when it comes to quality inspection.
Current Quality Inspection Methods
Now, diving into a little bit, what are customers doing today? Well, the easiest one of course is manual inspection. You can put a person in line right where the product is being developed or end of line. Now, there are some advantages and disadvantages of this. First, it's very easy to implement, but it is difficult and more importantly, expensive to scale. You're not able to cover necessarily a hundred percent of the objects, especially through hired throughput production lines and things like end of line inspection only really help once the entire process is done. And we know speaking from customers, the sooner that they're able to identify control defects, the better it's gonna be for the downstream processes and the less rework and waste is gonna come about. Now, there are customers who are interested in using machine vision to help automate and improve their processes. And there's a couple different ways that they're doing this today. One is they're trying to build in-house and that often requires a large team of data scientists of engineers to be able to design systems specifically tuned for different use cases. And that can be expensive to set up and maintain over time. On the flip side, they're also going to third parties and looking at ways to build systems to accomplish the same things. And while we know some of these third party companies do a really good job, they're often very use case specific, and they're often really purpose built for a specific production line. And those often are rule based and every time you have to change those rules. So if the environmental condition changes, if the lighting changes, if the product spec itself changes, that requires recalibration, updating of the rules and potentially higher and higher costs over time.
Overcoming Challenges with Amazon Lookout for Vision
So those are some of the challenges that we see with current options today and really some of the challenges that we see with customers getting started with computer vision in this space is around three core aspects, creating models, running models, and maintaining models. Starting at the first one, creating models, we know historically, computer vision and AIML models typically require thousands or tens of thousands of images to get to a high level of accuracy, we wanna reduce the barrier with Amazon Lookout for Vision. Customers without ML expertise can bring in as little as 30 images to train that initial model. And this allows customers to come in and try the service, see how the model's performing, and then be able to make a high business decision on whether they wanna continue with this or they want to add more data, or they want to deploy in production and begin to realize some cost savings. The second one is around running ML models. So we wanna make this easy by allowing customers to bring in their existing cameras their existing infrastructure, and in order to run Lookout for Vision, we don't have a hard requirement on the hardware and we want to make this as frictionless as possible. And we also have really strong integrations with related services like S3, SageMaker, GroundTruth, and other AWS services that can enable customers to use the different components within AWS to quickly run, host, and deploy their models in production. And the last one is around really maintaining the performance. If changes in the environmental conditions change, for example, we do offer a feedback mechanism which allows customers to look at the model predictions, verify whether they're accurate, whether they're actually seeing a normal or anomalous object, and use that feedback loop to feedback into the training data set where they can train a new model and make sure that the performance is relevant, it's customer representative, and that it meets their desired KPIs. So just on on the book ends of both those, right, it's a lower barrier of entry with 30 images. While we have confidence in our models that they can perform well, we know that it's not gonna work a 100% of the time for every single use case. And the way that we want to continue to incentivize customers to continue to use the service is through that feedback loop, improve the performance over time, and then you can reach a sufficient number of images that meets your production KPIs. So while we initially come in thinking we may need thousands or tens of thousands of images, that number may be much lower. And by using the feedback loop, you can actually identify what is the tipping point between reaching my goals and being able to stop collecting more data in order to improve my models. So Amazon Lookout for Vision was announced at Reinvents 2020 as a preview. We just launched as general available on February 24th, 2021. Very excited about being officially live. And to recap what it really is and how it works in one sentence, it's spots product defects using computer vision to automate quality inspection. Now I wanna dig into kind of the six core components where we see value for customers and they really fall into two different swim lanes. One is product quality and operational efficiency, and the other one is speed of deployment.
Enhancing Product Quality and Operational Efficiency
Talking about the first one, improving quality and production processes, our goal with Lookout for Vision is to help customers proactively identify anomalies and defects. We want them to have better data points that allows them to understand if they need to either improve their upstream processes to reduce defects or they need to maintain and improve the downstream aspects of this to make sure that at the end of the day, the highest quality product is going to their end consumers. Secondly, around reducing operational costs. So from a cost perspective, as we talked about earlier around warranties and scrap and rework, there's a real dollar value associated with every defective product. And by being able to identify and eventually quantify some of these costs through Lookout for Vision, we're trying to reduce that cost per unit. That allows you then to realize the bottom line savings. And third is around improve decision making. So thinking about if I want to identify a defect, what do I wanna do once that defect is identified, do I want to scrap that product? Do I wanna kick it off into a defective pile? Do I want a human to come in and verify whether it's an anomaly or not? Those are things that we're trying to enable. The best way to think about it is you can augment manual inspection where you have someone who's currently there and you're helping enabling them to better understand where the defect is or what decision needs to be made. Or you could fully automate the inspection and build in logic to kick off defective products. Whatever route is best suited for your use case, I think Lookout for Vision is able to help provide those data points for those decisions.
Accelerating Deployment and Innovation
Secondly, around speed of deployment, as I mentioned earlier, there is no ML experience required, as you'll see in the demo, it's a very user-friendly interface to upload images, train a model, deploy it. We want this to be a frictionless experience so that anyone from the VP of operations all the way down to the operator on the plant line could be enabled to drive projects, drive operational efficiencies, and be able to improve the quality of their production lines. Secondly, around identifying defects in diverse conditions. So we talked about earlier where historical third parties really focus on rule-based systems and they have to update and maintain those rules over time. The Lookout for Vision models behind the scenes have some robustness in them that they can actually work in diverse conditions in slight variations in product alignment, in slight variations in lighting. So those are things that we're trying to help mitigate and make sure that even with 30 images, customers can see a higher performing model. And lastly, around improving accuracy continuously, this is the feedback loop that I mentioned. If a model's performing at 90% and my KPI to use in production is 95% accuracy, we wanna be able to use things like the feedback loop, adding more images back that are verified and helps explain where the model is doing wrong and making sure that that insight is being filtered back into the next version of that model so that over time you get not only close to that accuracy performance benchmark, but also past it and continue to stay at that level.
Identifying Defects and Process Issues
So when we talk about the use cases around Lookout for Vision, there's three main use cases. The first one around detecting product defects. And the best way to think about this one is surface defects and shape defects. Surface defects can be anything from scratches, dents, holes, anything that you can kind of visually see with your eye right there. The shape defects are specific to things like soldering. If I'm expecting a square component and a rectangle component comes out, if it's slightly bent or misshaped, those are things that we can also find with Lookout for Vision. The second one is around identifying missing components. So think of this as an absence presence type of anomaly detection. If the circuit board on the left there has 35 components and I'm missing one of the components, I want to be able to flag that to the operator to be able to kick that off and make sure that that defective product doesn't reach an end consumer. And secondly, just like Pill Pack or another one of these pharmaceutical companies, I wanna make sure that not only have the right number of objects in the pack, but also the right kinds in there. So that's use case number two. So the third use case is around uncovering process issues. And this one is really around enabling our customers to have better insights to understand if their upstream processes are actually operating as expected. So there is logic with a Lookout for Vision. We have a dashboard where customers can go in and look at projects across different production lines and see how many defects are being identified and if there's a potential problem there. There is logic with other related services around like AWS CloudWatch for example, where you can create alerts that say, if I found five to 10 defects in the last 30 seconds, that's likely a red flag that there's a process issue here. So that's another use case.
Real-World Applications and Customer Success Stories
Now, just to take a step back, the industrial and manufacturing space is a very large and diverse space. Your use case may not fit cleanly into one of these three categories, and that's okay. The guiding principle that we're hearing from our science team is that if you can see a defect in an image with your human eye, that's something that our servers can pick up. So we're really interested in hearing about those use cases both within manufacturing but also outside too. As we can see this service help apply across different verticals and different industries that are not being discussed here today. Now a couple customer use cases, this one's a pretty fun one. So we have a customer called Dafgards. They're a household name in Sweden. They manufacture frozen pizzas. Dafgards came to us last year saying, we really want to figure out a better way to ensure our pizzas have enough cheese on them. We've seen that there's an inadequate amount of cheese. Customers want to make sure that the pizza itself has about 85 to 90% coverage in cheese. Can you help us out? And so we've been working with Dafgards behind the scenes and trying to build a model that helps them identify whether the pizza is adequately covered in cheese and has the correct toppings. So we've had a, we have yet to try the pizza, but we know that we're trying to help improve their quality and help maintain their brand name there in Sweden. For the second GE Healthcare, we've been working with them on a couple different use cases, particularly around their CT scan machines. Now, GE Healthcare today manufactures CT scan machines, and in order for them to really validate the quality of these machines, they put it through a variety of tests, capturing images, and then using a manual subject matter expert to go through the images and make sure that there's no defects present in any of them. Now that is one individual spending upwards of three hours per machine just manually and visually checking all images. GE Healthcare came to us wondering how can we help automate this? How can we help provide consistency? ## Introduction to Lookout for Vision How can we flag the most important anomalies so that our subject matter expert can only focus on those? And that's what we did with Lookout for Vision. We started building models with them last year and we're enabling them to deploy this in production and to be able to find defects so that their time to inspect has gone down from at least over an hour to just a few minutes just from a service like Lookout for Vision. So these are the value drivers that we're seeing across many different customers. Each has their own unique requirements, but we're excited that the service can be applied across different inspection points, across different verticals, across different use cases. And that's where we see Lookout for Vision really adding a lot of benefit for customers is that as they continue to grow and develop new production lines and new services, we want to grow with them and be able to provide a tool or service that allows them to scale in an automated fashion and allows them to meet their end requirements.
Customer Journey with Lookout for Vision
So the customer journey, and we'll definitely talk about this through the demo, but at a very high level, customers can bring in images to Lookout for Vision. It's a fully managed service where you can do all of your actions within the console itself. As they bring in images, they can create a project, create a data set, they can import those images into the data set itself. They can label them individually. We have a couple automated tools, which I'll touch on in the demo that helps them with labeling, but we also have labeling tools within the console itself. So you can do everything that you need to do directly in the console. There's no additional work that needs to be done outside of the service. Once the model is trained, you can deploy it, you can start it, start hosting it, you can deploy the model and you can start to run inference based on that model. And by being able to automate it, you can actually, like I mentioned, use alerts to deploy this in production and ultimately allow you to monitor it as images are being passed through the model and being run for inference. The continuous improvement mechanism, as it been mentioning is around the feedback loop and using the trial detections feature to make sure that you're validating your performance. And then the dashboard as a whole is where you monitor the defects and monitor the production lines across every single project.
Lookout for Vision Architecture and Future Plans
Now a quick pause around just the Lookout for Vision architecture. Some of this is aspirational, but this gives you kind of a quick overview of the connection points within Lookout for Vision and the broader AWS ecosystem within Cloud particular. So today it is a cloud only service. We do all the training and inference within cloud. We understand through customer conversations that Edge is in a very important part of this. So we are gonna be building different edge integration points in the future. AWS Panorama is one of the services that was also launched at Reinvent Panorama is essentially enables edge processing. And so we're working closely with them to see how do we bring Lookout for Vision models onto the Panorama device that can ultimately be set on site for a customer to allow them to run their inference at the edge. So those are the high level architecture that we plan to support today as well as what's coming in the future. But at a very high level, it's a fairly straightforward, easy to get started process as the service looks like today.
Amazon Lookout for Vision as Part of a Broader Solution
So the one thing I do want to call out is Amazon Lookout for Vision is part of a broader end to end solution. So from a customer perspective, they want cameras to help set up and capture images. They need a service to help process and identify the images for anomaly detection. And then they also wanna be able to integrate the output of that detection, whether that be good or bad into their upstreaming downstream processes. And so Amazon Lookout for Vision really is that middle component of just the image processing and anomaly detection, but we do have a partner ecosystem that helps with the other components. So Basler and ADLink, like for example, are our launch partners around camera hardware. We're working with them to do things like a POC kit to be able to help customers experiment with Lookout for Vision and make sure that they have the right camera specs to eventually move them to production. And then we also have a whole host of SI and consulting partners like Accenture and Deloitte who can help with the backend integration. We know there's a couple instances of how do I take the logic of Lookout for Vision. If I find an anomaly, how do I build an automated logic that kicks off that product into a defective pile? How do I integrate with the PLC to make sure that customers can trust that the output is actually gonna make sure that defective product gets pushed off somewhere else. So all those integrations are not something that we actively support we being AWS, but we do have this partner ecosystem that can help integrate that. And then we also have some technology solution partners like Foghorn and Tibco who can help us with some of the broader metrics. And can I visualize the outputs? Can I tie in the business metrics from other sensors or other products within the ITOT ecosystem at a customer location that make sure that the customer's getting the holistic view, not just a anomaly detection view, but how does that impact the other systems and services within the the shop floor.
Introduction to the Demo
So with that, I will jump to the demo. There's a couple things though I'll touch on over there. But the most important thing is that we wanna make this simple and easy to use as you'll hopefully see in this this video, we want to, within a few short clicks, you can bring in data, you can train an initial model and you can begin to evaluate the efficacy of that model for your specific use case.
Amazon Lookout for Vision Demo Overview
All right, so this is the Amazon Lookout for Vision demo. What you'll see is around, we're gonna kind of take a journey and pretend we're a circuit board manufacturer. And you see from the circuit board here, they have various types of defects. They have scratches on one side, they have another one with bent components that they wanna identify as well. And then there's also another one around soldering issues. So three very different types of anomalies, but all three very common in this space. And you see them quite frequently around circuit board manufacturers. So getting started, if I'm a customer who already has images, the first step I wanna do is go to the AWS consult, search for Amazon Lookout for Vision. As I get taken to Lookout for Vision, I can click get started. What we'll do is we create an S3 bucket for you, which holds hosts all the different assets and projects and data sets so you can cleanly understand where to go within the console. The first step is to create a project. Project name, the best way to think about this is this could be production line A or production line B or inspection point A and inspection point B, projects are really the hierarchy that helps separate between each individual inspection or use case that you're going after. Now, one of the ways that we want to enable better labeling or faster labeling for customers without the need to go through each individual image, we say 30 images is the bare minimum that still holds true. That may be very easy to label 30 images within the console. We know some customers have been doing this and want to bring in a thousand or 10,000 images and that makes it much more difficult for them to label those images individually. So what we want customers to do is bring in your images into S3, categorize them into two separate buckets, normal and anomaly. ## Introduction to Image Import and Labeling And the the way that we do that or the why we do that is as you'll see when we create a data set, which is step two, after creating a project, we have a function called import from S3 and we have an ability to actually label those two or import those images as well as simultaneously labeling those images for you. And so if I click on import images from S3, what I can do is I can copy in that location and I can say automatic labeling. And what that'll do is it'll read all the images in that bucket and folder that you just created. It'll read the two sub folders of normal and anomaly and it'll bring in those images and the labels accordingly. There's a few other features that you saw above, which I'll quickly touch on. Customers do have the option to bring in a single data set and we will do an automated split behind the scenes in order to have a training and test data set. Otherwise they can bring in their own. So if they have a very specific test data set that they want to benchmark against, we do offer that functionality.
Additional Image Import Options
The other two options to bring in images outside of S3 is uploading from a local drive, so that's one option. And then the third is around importing images labeled by Sagemaker GroundTruth. So if you have a broader annotation team, if you've actually used Sagemaker GroundTruth before and already labeled images along this line, you can actually just reference that manifest format so you don't have to rework at the end of the day. So as I bring in those images from S3 and I click Create Data Set, essentially what's gonna happen is those images are gonna be brought in and you'll see once I go to the dataset screen, it'll actually have all of those images imported with the respective labels already there. And this is really nice, as I mentioned, is the images already there, they already have their associated labels to it.
Training the Model
I can can already click Train Model. That's step one that makes it very simple for customers to go through and train a model. If I didn't have any of these images labeled, I can click add labels and I'll see a prompt that says normal or anomaly underneath each individual object. And that would essentially would allow you to adjust the labels if needed. Make sure you have a good ratio of normal to anomaly based on the right classification. And then I'll get a prompt that says you've labeled enough images to train a model. Again, the minimum benchmark is 30. Directionally speaking, we know the more images you provide likely the better the model's gonna perform. But we also know that collecting images, especially anomaly images can be difficult for customers as their defect rates might be very low. So this can be kind of step one to validate, see how the model's even performing and then make a decision whether our need to add more images.
So after a model is trained, we'll get some metrics around how it actually performed. And the key metrics that we really track here is precision recall and F1 score, the F1 being the overall model performance. You'll get some stats around the training duration. This one only take about 30 minutes on average it's anywhere up to two hours depending on the size of the dataset and the composition of the images themselves. But that is just to provide some feedback around how the model's performing and how quick it was trained.
In the bottom evaluation, what you'll see is whether that test dataset was auto split off in the beginning or if you gave an explicit test dataset when creating a data set, these are the results of how the model performed against those specific images. So what you can do is you can look at those images, you can look at whether it's predicted as normal or anomalous and the competence score associated with that. So that provides just some insights around how to actually evaluate where your model performs today.
Improving Model Accuracy
So hypothetically, if that model was let's say 85 to 90% and I had a benchmark of 95%, I have two options at this point. I could add more images as shown on the screen here and go back to the dataset screen, add more images as I collect them and then retrain a new model. The alternative option is running trial detection. And this is the feedback loop I keep talking about. Trial detection is a way to bring in images. You choose the model that was ultimately trained. You can bring in images from your local drive or you can bring in images from S3. So a very similar process at the end of the day that as you did from creating your initial training data set, as you bring in those images and run trial detection, what it does is it takes those images that your model has never seen and it's providing a prediction. It says here it's the normal, it's anomalous, here's our confidence score. And you as a user what you can do now as such a matter expert or another operator, you actually get the results that says, here's what we predicted. And you can go back and verify that.
Verifying and Adjusting Model Predictions
So we say on here we've predicted these five as anomaly, here's your confidence scores associated with it. You'll notice these are unverified 'cause we haven't actually as an operator gone and confirmed whether it's accurate. But what I can do is I can go to the top, I can click verify machine predictions, and this is totally personal preference or choice. You don't have to verify all of them, but whatever ones you do in particular, the ones that are incorrect. So if this one was incorrect, it should be normal. I definitely wanna click incorrect and then add those verified images back to the train data set. And what that does is then you'll see a notification shortly in the dataset that says five images have been added, five verified images have been added to my training dataset. I can go back and click on the data set. And before we had 30 or 60 images, excuse me, and now we have 65 images and you'll see the split of four of 'em we confirmed were in fact an anomaly, but that one that was actually supposed to be normal has been relabeled accordingly.
Continuous Model Improvement and Maintenance
So that's already automatically done for you. Those verified images are brought into the data set with the right labels. And at this point, what I can do now is train a second model and I can continue to train and improve that model until I reach that 95%. And this is the, not only helps you get to the right benchmark initially, but then also to maintain that model performance over time. If I start to see that it's not quite predicting my anomalies correctly, there could be some factors such as maybe the camera got bumped, maybe there's some environmental conditions that change that stirring off the model. So continuing to use that feedback loop can help maintain the performance of it. If something drastically changes the guidances, create a new project and almost start over. And that's because it's meant to help improve kind of the peripheral things. But if it's a completely different product, completely different camera production line, that will skew the model. So our guidance would be to just start a new project at that point.
Utilizing the Model
So when I am ready to use the model itself, we do offer, there's a use model tab within the the model sub folder or sub tab. We have code snippets already available for you to take that exact same model, start hosting it within Lookout for Vision and then calling the detect anomalies API to ultimately run the analysis. So what you'll see here is we can go through the CLI if you see everything we've done on the console can also be done through an API. So we have have a whole host of APIs available and you can see through the hosting of that model and then eventually calling the detect anomalies model here, the output is anomaly true or false and a confidence scorer. So we're telling the user is this particular image anomalous or not? And what's our confidence score associated with that? ## Service Overview Now using that output, this is where our partners, this is where other integration points solution architects can take that output and make it more meaningful. But that is the initial output that we're providing today as a service. It is an image level classification as normal anomaly and an associated confidence score. So coming back to the slide deck, really there, that's the end to end kind of overview of Lookout for Vision, you know we're helper, you're really excited about the service and interested in learning more and trying it yourself.
Learning Resources
We do have a lot of resources available to help enable some experimentation or just to learn more about the service itself. The first and foremost is our developer guide, which is available on our our broader product page. You go click on the docs, it talks through the step by step process of running all these models through the API or through the console itself. So I highly recommend folks to go check that out.
Free Tier Offer
The second call out I wanna talk about is our three month free tier. So we do offer 10 free hours of training per month and four free inference hours per month for the first three months. So that allows you really, to get your hands on training, couple models, understanding how to deploy it, trying to run inference. That's something that we wanna enable our customers to use as well as they get started in experimenting with Lookout for Vision, eventually talking about a POC and eventually using this in production.
GitHub Repository
And the last one is our Lookout for Vision GitHub repo. We newly launched this, this actually has a lot of good resources in there such as cloud formation templates. But the one call out i I will talk about is there is a circuit board dataset in their exact same one that you saw in the demo. We offer, we've already split out the images into train tests, trial detection, and then some extra images as well for additional experimentation. But encourage you all to download the data set, try to train your own model, understand how the service works and see what other applications this provides for your customers or other use cases out there.
Conclusion and Future Opportunities
And just to recap, right, Lookout for Vision. Here's the access to the console itself. We want to make sure that customers are aware of this and the value proposition we ultimately provide from a product quality and speed perspective. Hoping Lookout for Vision allows customers to get started quickly, allows them to realize some of the cost savings and more importantly, helps them mitigate against some of the challenges that we described earlier around not only tracking and quantifying costs, but also being able to help automate inspection processes and help them grow and scale faster than previously before. And that's really what we're excited about here at AWS. And in the Lookout for Vision team in particular, we see a ton of opportunity in this space and we look forward to working with you guys in the future on potential opportunities. And if there are questions, if there's use case opportunities, if there's follow up conversations around where we're going with the service, please do reach out and contact us. Really looking forward to having those discussions and especially if you've used the service, we love your feedback and how we can improve it as well. So for that, thank you for your time and I appreciate your attention today.
