Browse

Browse courses

Browse courses

PyTorch for Deep Learning Bootcamp

34. Transformer Encoder Overview.mp4

10. Section 08 PyTorch Paper Replicating · Video

Back Next lesson

Course content

14 chapters · 363 lessons

ZeroToMastery - PyTorch for Deep Learning Bootcamp Zero to Mastery 2025-4

Readme

1. Introduction

1. PyTorch for Deep Learning Bootcamp Zero to Mastery 2. Course Welcome and What Is Deep Learning 3. Exercise Meet Your Classmates and Instructor 4. Course Companion Book + Code + More 5. Machine Learning + Python Monthly 6. ZTM Plugin + Understanding Your Video Player 7. Set Your Learning Streak Goal

2. Section 00 PyTorch Fundamentals

1. Why Use Machine Learning or Deep Learning 2. The Number 1 Rule of Machine Learning and What Is Deep Learning Good For 3. Machine Learning vs. Deep Learning 4. Anatomy of Neural Networks 5. Different Types of Learning Paradigms 6. What Can Deep Learning Be Used For 7. What Is and Why PyTorch 8. What Are Tensors 9. What We Are Going To Cover With PyTorch 10. How To and How Not To Approach This Course 11. Important Resources For This Course 12. Getting Setup to Write PyTorch Code 13. Introduction to PyTorch Tensors 14. Creating Random Tensors in PyTorch 15. Creating Tensors With Zeros and Ones in PyTorch 16. Creating a Tensor Range and Tensors Like Other Tensors 17. Dealing With Tensor Data Types 18. Getting Tensor Attributes 19. Manipulating Tensors (Tensor Operations) 20. Matrix Multiplication (Part 1) 21. Matrix Multiplication (Part 2) The Two Main Rules of Matrix Multiplication 22. Matrix Multiplication (Part 3) Dealing With Tensor Shape Errors 23. Finding the Min Max Mean and Sum of Tensors (Tensor Aggregation) 24. Finding The Positional Min and Max of Tensors 25. Reshaping, Viewing and Stacking Tensors 26. Squeezing, Unsqueezing and Permuting Tensors 27. Selecting Data From Tensors (Indexing) 28. PyTorch Tensors and NumPy 29. PyTorch Reproducibility (Taking the Random Out of Random) 30. Different Ways of Accessing a GPU in PyTorch 31. Setting up Device Agnostic Code and Putting Tensors On and Off the GPU 32. PyTorch Fundamentals Exercises and Extra-Curriculum 33. Let's Have Some Fun (+ Free Resources)

3. Section 01 PyTorch Workflow

1. Introduction and Where You Can Get Help 2. Getting Setup and What We Are Covering 3. Creating a Simple Dataset Using the Linear Regression Formula 4. Splitting Our Data Into Training and Test Sets 5. Building a function to Visualize Our Data 6. Creating Our First PyTorch Model for Linear Regression 7. Breaking Down What's Happening in Our PyTorch Linear regression Model 8. Discussing Some of the Most Important PyTorch Model Building Classes 9. Checking Out the Internals of Our PyTorch Model 10. Making Predictions With Our Random Model Using Inference Mode 11. Training a Model Intuition (The Things We Need) 12. Setting Up an Optimizer and a Loss Function 13. PyTorch Training Loop Steps and Intuition 14. Writing Code for a PyTorch Training Loop 15. Reviewing the Steps in a Training Loop Step by Step 16. Running Our Training Loop Epoch by Epoch and Seeing What Happens 17. Writing Testing Loop Code and Discussing What's Happening Step by Step 18. Reviewing What Happens in a Testing Loop Step by Step 19. Writing Code to Save a PyTorch Model 20. Writing Code to Load a PyTorch Model 21. Setting Up to Practice Everything We Have Done Using Device-Agnostic Code 22. Putting Everything Together (Part 1) Data 23. Putting Everything Together (Part 2) Building a Model 24. Putting Everything Together (Part 3) Training a Model 25. Putting Everything Together (Part 4) Making Predictions With a Trained Model 26. Putting Everything Together (Part 5) Saving and Loading a Trained Model 27. PyTorch Workflow Exercises and Extra-Curriculum 28. Unlimited Updates

4. Section 02 PyTorch Neural Network Classification

1. Introduction to Machine Learning Classification With PyTorch 2. Classification Problem Example Input and Output Shapes 3. Typical Architecture of a Classification Neural Network (Overview) 4. Making a Toy Classification Dataset 5. Turning Our Data into Tensors and Making a Training and Test Split 6. Laying Out Steps for Modelling and Setting Up Device-Agnostic Code 7. Coding a Small Neural Network to Handle Our Classification Data 8. Making Our Neural Network Visual 9. Recreating and Exploring the Insides of Our Model Using nn.Sequential 10. Setting Up a Loss Function Optimizer and Evaluation Function for Our Classification Network 11. Going from Model Logits to Prediction Probabilities to Prediction Labels 12. Coding a Training and Testing Optimization Loop for Our Classification Model 13. Writing Code to Download a Helper Function to Visualize Our Models Predictions 14. Discussing Options to Improve a Model 15. Creating a New Model with More Layers and Hidden Units 16. Writing Training and Testing Code to See if Our New and Upgraded Model Performs Better 17. Creating a Straight Line Dataset to See if Our Model is Learning Anything 18. Building and Training a Model to Fit on Straight Line Data 19. Evaluating Our Models Predictions on Straight Line Data 20. Introducing the Missing Piece for Our Classification Model Non-Linearity 21. Building Our First Neural Network with Non-Linearity 22. Writing Training and Testing Code for Our First Non-Linear Model 23. Making Predictions with and Evaluating Our First Non-Linear Model 24. Replicating Non-Linear Activation Functions with Pure PyTorch 25. Putting It All Together (Part 1) Building a Multiclass Dataset 26. Creating a Multi-Class Classification Model with PyTorch 27. Setting Up a Loss Function and Optimizer for Our Multi-Class Model 28. Going from Logits to Prediction Probabilities to Prediction Labels with a Multi-Class Model 29. Training a Multi-Class Classification Model and Troubleshooting Code on the Fly 30. Making Predictions with and Evaluating Our Multi-Class Classification Model 31. Discussing a Few More Classification Metrics 32. PyTorch Classification Exercises and Extra-Curriculum 33. Course Check-In

5. Section 03 PyTorch Computer Vision

1. What Is a Computer Vision Problem and What We Are Going to Cover 2. Computer Vision Input and Output Shapes 3. What Is a Convolutional Neural Network (CNN) 4. Discussing and Importing the Base Computer Vision Libraries in PyTorch 5. Getting a Computer Vision Dataset and Checking Out Its- Input and Output Shapes 6. Visualizing Random Samples of Data 7. DataLoader Overview Understanding Mini-Batch 8. Turning Our Datasets Into DataLoaders 9. Model 0 Creating a Baseline Model with Two Linear Layers 10. Creating a Loss Function an Optimizer for Model 0 11. Creating a Function to Time Our Modelling Code 12. Writing Training and Testing Loops for Our Batched Data 13. Writing an Evaluation Function to Get Our Models Results 14. Setup Device-Agnostic Code for Running Experiments on the GPU 15. Model 1 Creating a Model with Non-Linear Functions 16. Model 1 Creating a Loss Function and Optimizer 17. Turing Our Training Loop into a Function 18. Turing Our Testing Loop into a Function 19. Training and Testing Model 1 with Our Training and Testing Functions 20. Getting a Results Dictionary for Model 1 21. Model 2 Convolutional Neural Networks High Level Overview 22. Model 2 Coding Our First Convolutional Neural Network with PyTorch 23. Model 2 Breaking Down Conv2D Step by Step 24. Model 2 Breaking Down MaxPool2D Step by Step 25. Model 2 Using a Trick to Find the Input and Output Shapes of Each of Our Layers 26. Model 2 Setting Up a Loss Function and Optimizer 27. Model 2 Training Our First CNN and Evaluating Its Results 28. Comparing the Results of Our Modelling Experiments 29. Making Predictions on Random Test Samples with the Best Trained Model 30. Plotting Our Best Model Predictions on Random Test Samples and Evaluating Them 31. Making Predictions Across the Whole Test Dataset and Importing Libraries to Plot a Confusion Matrix 32. Evaluating Our Best Models Predictions with a Confusion Matrix 33. Saving and Loading Our Best Performing Model 34. Recapping What We Have Covered Plus Exercises and Extra-Curriculum 35. Implement a New Life System

6. Section 04 PyTorch Custom Datasets

1. What Is a Custom Dataset and What We Are Going to Cover 2. Importing PyTorch and Setting Up Device-Agnostic Code 3. Downloading a Custom Dataset of Pizza, Steak and Sushi Images 4. Becoming One With the Data (Part 1) Exploring the Data Format 5. Becoming One With the Data (Part 2) Visualizing a Random Image 6. Becoming One With the Data (Part 3) Visualizing a Random Image with Matplotlib 7. Transforming Data (Part 1) Turning Images Into Tensors 8. Transforming Data (Part 2) Visualizing Transformed Images 9. Loading All of Our Images and Turning Them Into Tensors With ImageFolder 10. Visualizing a Loaded Image From the Train Dataset 11. Turning Our Image Datasets into PyTorch DataLoaders 12. Creating a Custom Dataset Class in PyTorch High Level Overview 13. Creating a Helper Function to Get Class Names From a Directory 14. Writing a PyTorch Custom Dataset Class from Scratch to Load Our Images 15. Compare Our Custom Dataset Class to the Original ImageFolder Class 16. Writing a Helper Function to Visualize Random Images from Our Custom Dataset 17. Turning Our Custom Datasets Into DataLoaders 18. Exploring State of the Art Data Augmentation With Torchvision Transforms 19. Building a Baseline Model (Part 1) Loading and Transforming Data 20. Building a Baseline Model (Part 2) Replicating Tiny VGG from Scratch 21. Building a Baseline Model (Part 3) Doing a Forward Pass to Test Our Model Shapes 22. Using the Torchinfo Package to Get a Summary of Our Model 23. Creating Training and Testing loop Functions 24. Creating a Train Function to Train and Evaluate Our Models 25. Training and Evaluating Model 0 With Our Training Functions 26. Plotting the Loss Curves of Model 0 27. Discussing the Balance Between Overfitting and Underfitting and How to Deal With Each 28. Creating Augmented Training Datasets and DataLoaders for Model 1 29. Constructing and Training Model 1 30. Plotting the Loss Curves of Model 1 31. Plotting the Loss Curves of All of Our Models Against Each Other 32. Predicting on Custom Data (Part 1) Downloading an Image 33. Predicting on Custom Data (Part2) Loading In a Custom Image With PyTorch 34. Predicting on Custom Data (Part 3) Getting Our Custom Image Into the Right Format 35. Predicting on Custom Data (Part 4) Turning Our Models Raw Outputs Into Prediction Labels 36. Predicting on Custom Data (Part 5) Putting It All Together 37. Summary of What We Have Covered Plus Exercises and Extra-Curriculum 38. Exercise Imposter Syndrome

7. Section 05 PyTorch Going Modular

1. What Is Going Modular and What We Are Going to Cover 2. Going Modular Notebook (Part 1) Running It End to End 3. Downloading a Dataset 4. Writing the Outline for Our First Python Script to Setup the Data 5. Creating a Python Script to Create Our PyTorch DataLoaders 6. Turning Our Model Building Code into a Python Script 7. Turning Our Model Training Code into a Python Script 8. Turning Our Utility Function to Save a Model into a Python Script 9. Creating a Training Script to Train Our Model in One Line of Code 10. Going Modular Summary, Exercises and Extra-Curriculum

8. Section 06 PyTorch Transfer Learning

1. Introduction What is Transfer Learning and Why Use It 2. Where Can You Find Pretrained Models and What We Are Going to Cover 3. Installing the Latest Versions of Torch and Torchvision 4. Downloading Our Previously Written Code from Going Modular 5. Downloading Pizza, Steak, Sushi Image Data from Github 6. Turning Our Data into DataLoaders with Manually Created Transforms 7. Turning Our Data into DataLoaders with Automatic Created Transforms 8. Which Pretrained Model Should You Use 9. Setting Up a Pretrained Model with Torchvision 10. Different Kinds of Transfer Learning 11. Getting a Summary of the Different Layers of Our Model 12. Freezing the Base Layers of Our Model and Updating the Classifier Head 13. Training Our First Transfer Learning Feature Extractor Model 14. Plotting the Loss Curves of Our Transfer Learning Model 15. Outlining the Steps to Make Predictions on the Test Images 16. Creating a Function Predict On and Plot Images 17. Making and Plotting Predictions on Test Images 18. Making a Prediction on a Custom Image 19. Main Takeaways, Exercises and Extra Curriculum

9. Section 07 PyTorch Experiment Tracking

1. What Is Experiment Tracking and Why Track Experiments 2. Getting Setup by Importing Torch Libraries and Going Modular Code 3. Creating a Function to Download Data 4. Turning Our Data into DataLoaders Using Manual Transforms 5. Turning Our Data into DataLoaders Using Automatic Transforms 6. Preparing a Pretrained Model for Our Own Problem 7. Setting Up a Way to Track a Single Model Experiment with TensorBoard 8. Training a Single Model and Saving the Results to TensorBoard 9. Exploring Our Single Models Results with TensorBoard 10. Creating a Function to Create SummaryWriter Instances 11. Adapting Our Train Function to Be Able to Track Multiple Experiments 12. What Experiments Should You Try 13. Discussing the Experiments We Are Going to Try 14. Downloading Datasets for Our Modelling Experiments 15. Turning Our Datasets into DataLoaders Ready for Experimentation 16. Creating Functions to Prepare Our Feature Extractor Models 17. Coding Out the Steps to Run a Series of Modelling Experiments 18. Running Eight Different Modelling Experiments in 5 Minutes 19. Viewing Our Modelling Experiments in TensorBoard 20. Loading In the Best Model and Making Predictions on Random Images from the Test Set 21. Making a Prediction on Our Own Custom Image with the Best Model 22. Main Takeaways, Exercises and Extra Curriculum

10. Section 08 PyTorch Paper Replicating

1. What Is a Machine Learning Research Paper 2. Why Replicate a Machine Learning Research Paper 3. Where Can You Find Machine Learning Research Papers and Code 4. What We Are Going to Cover 5. Getting Setup for Coding in Google Colab 6. Downloading Data for Food Vision Mini 7. Turning Our Food Vision Mini Images into PyTorch DataLoaders 8. Visualizing a Single Image 9. Replicating a Vision Transformer - High Level Overview 10. Breaking Down Figure 1 of the ViT Paper 11. Breaking Down the Four Equations Overview and a Trick for Reading Papers 12. Breaking Down Equation 1 13. Breaking Down Equations 2 and 3 14. Breaking Down Equation 4 15. Breaking Down Table 1 16. Calculating the Input and Output Shape of the Embedding Layer by Hand 17. Turning a Single Image into Patches (Part 1 Patching the Top Row) 18. Turning a Single Image into Patches (Part 2 Patching the Entire Image) 19. Creating Patch Embeddings with a Convolutional Layer 20. Exploring the Outputs of Our Convolutional Patch Embedding Layer 21. Flattening Our Convolutional Feature Maps into a Sequence of Patch Embeddings 22. Visualizing a Single Sequence Vector of Patch Embeddings 23. Creating the Patch Embedding Layer with PyTorch 24. Creating the Class Token Embedding 25. Creating the Class Token Embedding - Less Birds 26. Creating the Position Embedding 27. Equation 1 Putting it All Together 28. Equation 2 Multihead Attention Overview 29. Equation 2 Layernorm Overview 30. Turning Equation 2 into Code 31. Checking the Inputs and Outputs of Equation 32. Equation 3 Replication Overview 33. Turning Equation 3 into Code 34. Transformer Encoder Overview NOW 35. Combining Equation 2 and 3 to Create the Transformer Encoder 36. Creating a Transformer Encoder Layer with In-Built PyTorch Layer 37. Bringing Our Own Vision Transformer to Life - Part 1 Gathering the Pieces of the Puzzle 38. Bringing Our Own Vision Transformer to Life - Part 2 Putting Together the Forward Method 39. Getting a Visual Summary of Our Custom Vision Transformer 40. Creating a Loss Function and Optimizer from the ViT Paper 41. Training our Custom ViT on Food Vision Mini 42. Discussing what Our Training Setup Is Missing 43. Plotting a Loss Curve for Our ViT Model 44. Getting a Pretrained Vision Transformer from Torchvision and Setting it Up 45. Preparing Data to Be Used with a Pretrained ViT 46. Training a Pretrained ViT Feature Extractor Model for Food Vision Mini 47. Saving Our Pretrained ViT Model to File and Inspecting Its Size 48. Discussing the Trade-Offs Between Using a Larger Model for Deployments 49. Making Predictions on a Custom Image with Our Pretrained ViT 50. PyTorch Paper Replicating Main Takeaways, Exercises and Extra-Curriculum

11. Section 09 PyTorch Model Deployment

1. What is Machine Learning Model Deployment and Why Deploy a Machine Learning Model 2. Three Questions to Ask for Machine Learning Model Deployment 3. Where Is My Model Going to Go 4. How Is My Model Going to Function 5. Some Tools and Places to Deploy Machine Learning Models 6. What We Are Going to Cover 7. Getting Setup to Code 8. Downloading a Dataset for Food Vision Mini 9. Outlining Our Food Vision Mini Deployment Goals and Modelling Experiments 10. Creating an EffNetB2 Feature Extractor Model 11. Create a Function to Make an EffNetB2 Feature Extractor Model and Transforms 12. Creating DataLoaders for EffNetB2 13. Training Our EffNetB2 Feature Extractor and Inspecting the Loss Curves 14. Saving Our EffNetB2 Model to File 15. Getting the Size of Our EffNetB2 Model in Megabytes 16. Collecting Important Statistics and Performance Metrics for Our EffNetB2 Model 17. Creating a Vision Transformer Feature Extractor Model 18. Creating DataLoaders for Our ViT Feature Extractor Model 19. Training Our ViT Feature Extractor Model and Inspecting Its Loss Curves 20. Saving Our ViT Feature Extractor and Inspecting Its Size 21. Collecting Stats About Our ViT Feature Extractor 22. Outlining the Steps for Making and Timing Predictions for Our Models 23. Creating a Function to Make and Time Predictions with Our Models 24. Making and Timing Predictions with EffNetB2 25. Making and Timing Predictions with ViT 26. Comparing EffNetB2 and ViT Model Statistics 27. Visualizing the Performance vs Speed Trade-off 28. Gradio Overview and Installation 29. Gradio Function Outline 30. Creating a Predict Function to Map Our Food Vision Mini Inputs to Outputs 31. Creating a List of Examples to Pass to Our Gradio Demo 32. Bringing Food Vision Mini to Life in a Live Web Application 33. Getting Ready to Deploy Our App Hugging Face Spaces Overview 34. Outlining the File Structure of Our Deployed App 35. Creating a Food Vision Mini Demo Directory to House Our App Files 36. Creating an Examples Directory with Example Food Vision Mini Images 37. Writing Code to Move Our Saved EffNetB2 Model File 38. Turning Our EffNetB2 Model Creation Function Into a Python Script 39. Turning Our Food Vision Mini Demo App Into a Python Script 40. Creating a Requirements File for Our Food Vision Mini App 41. Downloading Our Food Vision Mini App Files from Google Colab 42. Uploading Our Food Vision Mini App to Hugging Face Spaces Programmatically 43. Running Food Vision Mini on Hugging Face Spaces and Trying it Out 44. Food Vision Big Project Outline 45. Preparing an EffNetB2 Feature Extractor Model for Food Vision Big 46. Downloading the Food 101 Dataset 47. Creating a Function to Split Our Food 101 Dataset into Smaller Portions 48. Turning Our Food 101 Datasets into DataLoaders 49. Training Food Vision Big Our Biggest Model Yet! 50. Outlining the File Structure for Our Food Vision Big 51. Downloading an Example Image and Moving Our Food Vision Big Model File 52. Saving Food 101 Class Names to a Text File and Reading them Back In 53. Turning Our EffNetB2 Feature Extractor Creation Function into a Python Script 54. Creating an App Script for Our Food Vision Big Model Gradio Demo 55. Zipping and Downloading Our Food Vision Big App Files 56. Deploying Food Vision Big to Hugging Face Spaces 57. PyTorch Mode Deployment Main Takeaways, Extra-Curriculum and Exercises

12. Introduction to PyTorch 2.0 and torch.compile

1. Introduction to PyTorch 2.0 2. What We Are Going to Cover and PyTorch 2 Reference Materials 3. Getting Started with PyTorch 2.0 in Google Colab 4. PyTorch 2.0 - 30 Second Intro 5. Getting Setup for PyTorch 2.0 6. Getting Info from Our GPUs and Seeing if They're Capable of Using PyTorch 2.0 7. Setting the Default Device in PyTorch 2.0 8. Discussing the Experiments We Are Going to Run for PyTorch 2.0 9. Creating a Function to Setup Our Model and Transforms 10. Discussing How to Get Better Relative Speedups for Training Models 11. Setting the Batch Size and Data Size Programmatically 12. Getting More Speedups with TensorFloat-32 13. Downloading the CIFAR10 Dataset 14. Creating Training and Test DataLoaders 15. Preparing Training and Testing Loops with Timing Steps 16. Experiment 1 - Single Run without Torch Compile 17. Experiment 2 - Single Run with Torch Compile 18. Comparing the Results of Experiments 1 and 2 19. Saving the Results of Experiments 1 and 2 20. Preparing Functions for Experiments 3 and 4 21. Experiment 3 - Training a Non-Compiled Model for Multiple Runs 22. Experiment 4 - Training a Compiled Model for Multiple Runs 23. Comparing the Results of Experiments 3 and 4 24. Potential Extensions and Resources to Learn More

13. Where To Go From Here

1. Thank You! 2. Review This Course! 3. Become An Alumni 4. Learning Guideline 5. ZTM Events Every Month 6. LinkedIn Endorsements

Free courses from the real sources — indexed, organized, and searchable. No accounts, no paywalls.

Popular topics

Python
JavaScript
Data Science
Machine Learning

© 2026 CourseDen. All courses belong to their original authors.

Always free · No accounts required