Get Started

A Guide to Deploying YOLO26 on Amazon SageMaker Endpoints

Amazon SageMaker Endpoints provide managed, scalable hosting for computer vision models like Ultralytics YOLO26 in production machine learning applications. A successful model deployment depends on understanding the setup, configuration, and deployment steps on both the model and the AWS side.

This guide will take you through the process of deploying YOLO26 PyTorch models on Amazon SageMaker Endpoints step by step. You'll learn the essentials of preparing your AWS environment, configuring the model appropriately, and using tools like AWS CloudFormation and the AWS Cloud Development Kit (CDK) for deployment.

Amazon SageMaker#

Amazon SageMaker ML platform architecture

Amazon SageMaker is a machine learning service from Amazon Web Services (AWS) that simplifies the process of building, training, and deploying machine learning models. It provides a broad range of tools for handling various aspects of machine learning workflows. This includes automated features for tuning models, options for training models at scale, and straightforward methods for deploying models into production. SageMaker supports popular machine learning frameworks, offering the flexibility needed for diverse projects. Its features also cover data labeling, workflow management, and performance analysis.

Deploying YOLO26 on Amazon SageMaker Endpoints#

Deploying YOLO26 on Amazon SageMaker lets you use its managed environment for real-time inference and take advantage of features like autoscaling. Take a look at the AWS architecture below.

AWS architecture for hosting YOLO on an Amazon SageMaker endpoint

Step 1: Set Up Your AWS Environment#

First, ensure you have the following prerequisites in place:

  • An AWS Account: If you don't already have one, sign up for an AWS account.

  • Configured IAM Roles: You'll need an IAM role with the necessary permissions for Amazon SageMaker, AWS CloudFormation, and Amazon S3. This role should have policies that allow it to access these services.

  • AWS CLI: If not already installed, download and install the AWS Command Line Interface (CLI) and configure it with your account details. Follow the AWS CLI instructions for installation.

  • AWS CDK: If not already installed, install the AWS Cloud Development Kit (CDK), which will be used for scripting the deployment. Follow the AWS CDK instructions for installation.

  • Adequate Service Quota: Confirm that you have sufficient quotas for two separate resources in Amazon SageMaker: one for ml.m5.4xlarge for endpoint usage and another for ml.m5.4xlarge for notebook instance usage. Each of these requires a minimum of one quota value. If your current quotas are below this requirement, it's important to request an increase for each. You can request a quota increase by following the detailed instructions in the AWS Service Quotas documentation.

Step 2: Clone the YOLO26 SageMaker Repository#

The next step is to clone the specific AWS repository that contains the resources for deploying YOLO26 on SageMaker. This repository, hosted on GitHub, includes the necessary CDK scripts and configuration files.

  • Clone the GitHub Repository: Execute the following command in your terminal to clone the host-yolov8-on-sagemaker-endpoint repository:

    git clone https://github.com/aws-samples/host-yolov8-on-sagemaker-endpoint.git
  • Navigate to the Cloned Directory: Change your directory to the cloned repository:

    cd host-yolov8-on-sagemaker-endpoint/yolov8-pytorch-cdk

Step 3: Set Up the CDK Environment#

Now that you have the necessary code, set up your environment for deploying with AWS CDK.

  • Create a Python Virtual Environment: This isolates your Python environment and dependencies. Run:

    python3 -m venv .venv
  • Activate the Virtual Environment:

    source .venv/bin/activate
  • Install Dependencies: Install the required Python dependencies for the project:

    pip3 install -r requirements.txt
  • Upgrade AWS CDK Library: Ensure you have the latest version of the AWS CDK library:

    pip install --upgrade aws-cdk-lib

Step 4: Create the AWS CloudFormation Stack#

  • Synthesize the CDK Application: Generate the AWS CloudFormation template from your CDK code:

    cdk synth
  • Bootstrap the CDK Application: Prepare your AWS environment for CDK deployment:

    cdk bootstrap
  • Deploy the Stack: This will create the necessary AWS resources and deploy your model:

    cdk deploy

Step 5: Deploy the YOLO Model#

After creating the AWS CloudFormation Stack, the next step is to deploy YOLO26.

  • Open the Notebook Instance: Go to the AWS Console and navigate to the Amazon SageMaker service. Select "Notebook Instances" from the dashboard, then locate the notebook instance that was created by your CDK deployment script. Open the notebook instance to access the Jupyter environment.

  • Access and Modify inference.py: After opening the SageMaker notebook instance in Jupyter, locate the inference.py file. Edit the output_fn function in inference.py as shown below and save your changes to the script, ensuring that there are no syntax errors.

    import json
    
    def output_fn(prediction_output, content_type):
        """Formats model outputs as JSON string, extracting attributes like boxes, masks, keypoints."""
        print("Executing output_fn from inference.py ...")
        infer = {}
        for result in prediction_output:
            if result.boxes is not None:
                infer["boxes"] = result.boxes.numpy().data.tolist()
            if result.masks is not None:
                infer["masks"] = result.masks.numpy().data.tolist()
            if result.keypoints is not None:
                infer["keypoints"] = result.keypoints.numpy().data.tolist()
            if result.obb is not None:
                infer["obb"] = result.obb.numpy().data.tolist()
            if result.probs is not None:
                infer["probs"] = result.probs.numpy().data.tolist()
        return json.dumps(infer)
  • Deploy the Endpoint Using 1_DeployEndpoint.ipynb: In the Jupyter environment, open the 1_DeployEndpoint.ipynb notebook located in the sm-notebook directory. Set model_name in the notebook to a YOLO26 checkpoint such as yolo26n.pt (it defaults to a YOLOv8 model), then run the cells to download the model, package it with the updated inference code, and upload it to an Amazon S3 bucket. The notebook will guide you through creating and deploying a SageMaker endpoint for the YOLO26 model.

Step 6: Testing Your Deployment#

Now that your YOLO26 model is deployed, it's important to test its performance and functionality.

  • Open the Test Notebook: In the same Jupyter environment, locate and open the 2_TestEndpoint.ipynb notebook, also in the sm-notebook directory.

  • Run the Test Notebook: Follow the instructions within the notebook to test the deployed SageMaker endpoint. This includes sending an image to the endpoint and running inferences. Then, you'll plot the output to visualize the model's performance and accuracy, as shown below.

YOLO26 detection results returned by the SageMaker test endpoint

  • Clean-Up Resources: The test notebook will also guide you through the process of cleaning up the endpoint and the hosted model. This is an important step to manage costs and resources effectively, especially if you do not plan to use the deployed model immediately.

Step 7: Monitoring and Management#

After testing, continuous monitoring and management of your deployed model are essential.

  • Monitor with Amazon CloudWatch: Regularly check the performance and health of your SageMaker endpoint using Amazon CloudWatch.

  • Manage the Endpoint: Use the SageMaker console for ongoing management of the endpoint. This includes scaling, updating, or redeploying the model as required.

By completing these steps, you will have deployed and tested a YOLO26 model on Amazon SageMaker Endpoints. The same workflow applies to deploying other models on AWS.

Summary#

This guide took you step by step through deploying YOLO26 on Amazon SageMaker Endpoints using AWS CloudFormation and the AWS Cloud Development Kit (CDK). The process includes cloning the AWS sample repository from the AWS Machine Learning Blog post on hosting YOLOv8 PyTorch models on Amazon SageMaker Endpoints, setting up the CDK environment, deploying the model using AWS services, and testing its performance on SageMaker.

FAQ#

  • To deploy the Ultralytics YOLO26 model on Amazon SageMaker Endpoints, follow these steps:

    1. Set Up Your AWS Environment: Ensure you have an AWS Account, IAM roles with necessary permissions, and the AWS CLI configured. Install AWS CDK if not already done (refer to the AWS CDK instructions).
    2. Clone the YOLO26 SageMaker Repository:
      git clone https://github.com/aws-samples/host-yolov8-on-sagemaker-endpoint.git
      cd host-yolov8-on-sagemaker-endpoint/yolov8-pytorch-cdk
    3. Set Up the CDK Environment: Create a Python virtual environment, activate it, install dependencies, and upgrade AWS CDK library.
      python3 -m venv .venv
      source .venv/bin/activate
      pip3 install -r requirements.txt
      pip install --upgrade aws-cdk-lib
    4. Deploy using AWS CDK: Synthesize and deploy the CloudFormation stack, bootstrap the environment.
      cdk synth
      cdk bootstrap
      cdk deploy
  • To deploy YOLO26 on Amazon SageMaker, ensure you have the following prerequisites:

    1. AWS Account: Active AWS account (sign up for AWS).
    2. IAM Roles: Configured IAM roles with permissions for SageMaker, CloudFormation, and Amazon S3.
    3. AWS CLI: Installed and configured AWS Command Line Interface (AWS CLI installation guide).
    4. AWS CDK: Installed AWS Cloud Development Kit (CDK setup guide).
    5. Service Quotas: Sufficient quotas for ml.m5.4xlarge instances for both endpoint and notebook usage (request a quota increase).
  • Using Ultralytics YOLO26 on Amazon SageMaker offers several advantages:

    1. Scalability and Management: SageMaker provides a managed environment with features like autoscaling, which helps in real-time inference needs.
    2. Integration with AWS Services: Integrate with other AWS services, such as S3 for data storage, CloudFormation for infrastructure as code, and CloudWatch for monitoring.
    3. Ease of Deployment: Simplified setup using AWS CDK scripts and streamlined deployment processes.
    4. Performance: Use Amazon SageMaker's managed infrastructure to run large-scale inference workloads.
  • Yes, you can customize the inference logic for YOLO26 on Amazon SageMaker:

    1. Modify inference.py: Locate and customize the output_fn function in the inference.py file to tailor output formats.

      import json
      
      def output_fn(prediction_output, content_type):
          """Formats model outputs as JSON string, extracting attributes like boxes, masks, keypoints."""
          infer = {}
          for result in prediction_output:
              if result.boxes is not None:
                  infer["boxes"] = result.boxes.numpy().data.tolist()
              # Add more processing logic if necessary
          return json.dumps(infer)
    2. Deploy Updated Model: Ensure you redeploy the model using Jupyter notebooks provided (1_DeployEndpoint.ipynb) to include these changes.

    Refer to the detailed steps for deploying the modified model.

  • To test the deployed YOLO26 model on Amazon SageMaker:

    1. Open the Test Notebook: Locate the 2_TestEndpoint.ipynb notebook in the SageMaker Jupyter environment.
    2. Run the Notebook: Follow the notebook's instructions to send an image to the endpoint, perform inference, and display results.
    3. Visualize Results: Use the notebook's plotting code to draw the predicted bounding boxes on the test image.

    For comprehensive testing instructions, visit the testing section.

Comments10

Ask a question, share your results, or help someone else.

  1. User

    I want to upload the YOLOv8 model I trained on my own dataset to Jupyter, and then compress it into a model.tar.gz file following the official deployendpoint steps. However, when I use testendpoint, I encounter an error. If I rename my trained model to yolov8, the test succeeds, but it uses the original model from the official website instead of my custom one. How should I modify inference.py to ensure it loads my trained model?

    1
    1. User

      Thank you for your answer. I had followed the 1_DeployEndpoint.ipynb to delpoy my own model. In the 1.4 Create the SageMaker PyTorchModel, I had set the env={'TS_MAX_RESPONSE_SIZE':'20000000', 'MY_MODEL': 'my_own_model'}. And in the inference.py:

      def model_fn(model_dir):
          print("Executing model_fn from inference.py ...")
          env = os.environ
          model = YOLO(os.path.join(model_dir, env['MY_MODEL']))
          return model

      I think it will be use my own model directly. But it failed, and it occur this problem:

      ModelError: An error occurred (ModelError) when calling the InvokeEndpoint operation: Received server error (0)from primary with message "Your invocation timed out while waiting for a response from container primary. Reviewthe latency metrics for each container in Amazon Cloudatch, resolve the issue, and try again.". See https://useast-1. console. aws. aazon. com/cloudwatch/home?region=us-east-1#logEventViewer: group=/aws/sagemaker Endpoints/yolov8-pytorch-2024-09-06-11-33-49-418696 in account 211125332801 for more information.

      Could you tell me how to modify the inference.py to load my own model. Thank you so much.

    2. Glenn JocherUltralytics

      To ensure your custom YOLOv8 model is used, verify that the model_fn in inference.py correctly points to your model's path. Ensure the MY_MODEL environment variable matches your model's filename. If issues persist, check CloudWatch logs for detailed error messages. For further guidance, refer to the Ultralytics documentation on Amazon SageMaker.

  2. SJTUPandaGitHub

    Thanks for your sharing, it's very clear and useful. My own deployment workflow on AWS has already built but the test result is not so satisfying. The performance of detection result of yolo11-xl on AWS is worse than on my own laptop. I not sure if there is some default quantization when it's deployed to the AWS(I did not modified much to host-yolov8-on-sagemaker-endpoint), or do you have some other ideas about which would bring this significant performance differences?

    1. Glenn JocherUltralytics

      @SJTUPanda thank you for your feedback! The performance difference could stem from several factors like hardware disparities (AWS instance vs laptop GPU), inference settings, or environment configurations. First, verify that your SageMaker instance type matches your local hardware capabilities, and ensure identical inference parameters (e.g., imgsz, batch). Check our Amazon SageMaker deployment guide for optimization tips and consider profiling both environments with yolo val to isolate bottlenecks. Let us know if specific metrics differ! 🚀

    2. SJTUPandaGitHub

      Thanks for your quick reply. The issue is fixed after I comment out resize related code. Although the model is trained with (640,640) and the input image size is (720,440), using the raw images directly as input is better than resized it. I am curious the input images would be auto-resized by the model just as in the training/testing part? And why resized by myself has so much impact on the results? If these part code should be update because yolo version updating?

      image_height, image_width, _ = orig_image.shape
      model_height, model_width = 640, 640
      x_ratio = image_width/model_width
      y_ratio = image_height/model_height
      
      # Resize the image as numpy array
      resized_image = cv2.resize(orig_image, (model_height, model_width))
      # Conver the array into jpeg
      resized_jpeg = cv2.imencode('.jpg', resized_image)[1]
      # Serialize the jpg using base 64
      payload = base64.b64encode(resized_jpeg).decode('utf-8')
    3. Glenn JocherUltralytics

      Thank you for sharing your findings! YOLO models do automatically resize inputs to match their training dimensions while preserving aspect ratio with letterboxing. Manual resizing can distort aspect ratios or introduce artifacts, potentially degrading performance. We recommend using the model's native preprocessing by feeding full-size images directly, as this maintains proper scaling and aspect ratio handling. For implementation details, refer to our model deployment best practices regarding input consistency.

    1. Glenn JocherUltralytics

      Olá! 👋 Os recursos mencionados para implementar o YOLOv8 no Amazon SageMaker não são pagos por si só. No entanto, o uso do Amazon SageMaker e outros serviços da AWS associados à implementação e execução do modelo podem incorrer em custos conforme o uso. A AWS oferece uma camada gratuita que você pode aproveitar para experimentar alguns serviços sem custo, mas é sempre bom verificar a página de preços do SageMaker para detalhes específicos. Se precisar de ajuda para estimar os custos ou tiver outras dúvidas, estou aqui para ajudar! 🚀