Skip to main content

Rollback Process - API

This guide provides step-by-step instructions for rolling back API services to previous versions when issues are discovered in production or staging environments. The rollback process allows you to rollback individual services independently.

Prerequisites​

Before starting the rollback process, ensure you have:

  • Access to the repository and GitHub Actions
  • Appropriate permissions for the target environment
  • Knowledge of the version to rollback to
  • Understanding of which service(s) need to be rolled back
  • Understanding of service dependencies
warning

Rollback should only be performed when necessary. Consider deploying a hotfix first if the issues can be resolved quickly. Be aware of service dependencies when rolling back individual services.

Rollback Process​

Follow these steps to rollback API services to a previous version:

Step 1: Identify Target Version and Service​

  1. Review Version History

    • Identify the last known stable version for the affected service
    • Verify the version number (e.g., 1.0.0, 1.2.3)
    • Confirm the version exists in the container registry

  2. Identify Affected Service(s)

    • Determine which API service(s) need to be rolled back
    • Consider service dependencies (some services may depend on others)
    • Verify service names match the deployment configuration

  3. Verify Version Compatibility

    • Ensure the target version is compatible with the current environment
    • Check for any database schema requirements
    • Verify API contract compatibility with other services
    • Check for breaking changes in the API
Service Dependencies

When rolling back a service, ensure that:

  • Other services are compatible with the rolled-back version
  • API contracts remain compatible
  • Database schemas are compatible
  • No breaking changes affect dependent services

Step 2: Navigate to GitHub Actions​

  1. Access Repository

    • Go to the repository on GitHub
    • Ensure you're viewing the correct repository and branch

  2. Open Actions Tab

    • Click on the Actions tab in the repository navigation
    • Locate the Prod Pipeline for Manual Deployment (each service) workflow in the workflow list

Step 3: Initiate Rollback Workflow​

  1. Run Workflow

    • Click on Prod Pipeline for Manual Deployment (each service) to open the workflow
    • Click the Run workflow dropdown button (located on the right side)
    • Select Run workflow from the dropdown

  2. Configure Rollback Parameters

    • Version: Enter the version number you want to rollback to (e.g., 1.0.0)
      • Use semantic versioning format: x.y.z
      • Ensure the version exists in the container registry
    • Service: Select the service you want to rollback from the dropdown
      • Choose the specific service that needs to be rolled back
      • Verify the service name matches the deployment configuration
    • Branch: Select the appropriate branch (usually main for production)

  3. Confirm Rollback

    • Review all parameters carefully
    • Double-check the service name and version
    • Click Run workflow to initiate the rollback process
Version Format

The version should follow semantic versioning (SemVer) format:

  • Example: 1.0.0, 1.2.3, 2.0.1
  • Do not include prefixes like v or version
Multiple Services

If multiple services need to be rolled back:

  1. Rollback services one at a time
  2. Verify each service after rollback before proceeding
  3. Consider service dependencies when determining rollback order
  4. Start with services that have no dependencies, then proceed to dependent services

Step 4: Monitor Rollback Process​

  1. Track Workflow Execution

    • Monitor the workflow run in the Actions tab
    • Watch for any errors or warnings during execution
    • Check each step of the deployment pipeline

  2. Verify Deployment Steps

    • Container build (if applicable)
    • Image push to registry
    • Container deployment
    • Health checks
    • Service registration (if applicable)

Step 5: Verify Rollback Success​

After the workflow completes, verify the rollback was successful:

  1. Check Service Status

    • Verify the API service is running
    • Check that the service container is healthy
    • Monitor application logs for errors

  2. Test API Endpoints

    • Test critical API endpoints
    • Verify authentication and authorization
    • Check response formats and data integrity
    • Test error handling

  3. Verify Service Integration

    • Test integration with dependent services
    • Verify API contracts are compatible
    • Check that other services can communicate with the rolled-back service

  4. Monitor System Health

    • Check application metrics and performance
    • Review error logs for any issues
    • Monitor resource usage (CPU, memory, disk)
    • Verify database connections and queries
Verification Checklist
  • Service container is running and healthy
  • API endpoints respond correctly
  • Authentication and authorization work
  • No critical errors in logs
  • Performance is within acceptable range
  • Integration with other services works
  • Database connections are stable

Rolling Back Multiple Services​

If multiple services need to be rolled back:

  1. Determine Rollback Order

    • Identify service dependencies
    • Rollback independent services first
    • Then rollback dependent services

  2. Rollback One Service at a Time

    • Complete rollback and verification for one service
    • Verify integration before proceeding to the next service
    • Document the rollback order for reference

  3. Verify Overall System

    • After all services are rolled back, test the entire system
    • Verify end-to-end workflows
    • Check system integration and data flow

Post-Rollback Actions​

After a successful rollback:

  1. Document the Rollback

    • Record the reason for rollback
    • Document the service(s) rolled back
    • Note the version rolled back from and to
    • Record any issues encountered during rollback

  2. Communicate with Team

    • Notify the team about the rollback
    • Share the reason, affected services, and target versions
    • Update relevant documentation

  3. Investigate Root Cause

    • Analyze why the deployment caused issues
    • Review code changes between versions
    • Plan fixes for the problematic version

  4. Plan Forward

    • Decide on next steps (hotfix, re-deployment, etc.)
    • Schedule fixes for the rolled-back version
    • Update deployment procedures if needed

Troubleshooting​

Common Issues​

Workflow fails to start

  • Verify you have permissions to run workflows
  • Check that the workflow file exists and is valid
  • Ensure you're on the correct branch

Version not found

  • Verify the version exists in the container registry
  • Check the version format matches exactly
  • Ensure the version was previously deployed for the selected service

Service not found in dropdown

  • Verify the service name matches the deployment configuration
  • Check that the service has been deployed before
  • Ensure the service is configured in the workflow

Rollback deployment fails

  • Check workflow logs for specific error messages
  • Verify container registry access and permissions
  • Ensure all required secrets and environment variables are configured
  • Check for resource constraints (disk space, memory)
  • Verify nginx configuration if routing is affected

Service not accessible after rollback

  • Verify the service container is running: docker ps
  • Check service logs: docker logs <container-name>
  • Verify environment variables are correctly set
  • Ensure firewall rules allow traffic on the configured ports
  • Check nginx routing configuration (if applicable)
  • Verify service registration and discovery (if applicable)

API compatibility issues

  • Check API contract compatibility with other services
  • Verify request/response formats match expectations
  • Review API versioning and breaking changes
  • Test integration with dependent services

Database compatibility issues

  • Verify database schema compatibility
  • Check for missing or incompatible migrations
  • Review database connection settings
  • Verify data integrity after rollback

Performance issues after rollback

  • Check resource usage (CPU, memory, disk)
  • Review application logs for performance-related errors
  • Verify database connection pool settings
  • Check for memory leaks or resource exhaustion
  • Review nginx configuration and routing

Best Practices​

  1. Maintain Version History - Keep detailed records of all deployed versions per service
  2. Test Rollback Procedures - Periodically test rollback in staging environment
  3. Document Service Dependencies - Maintain documentation of service dependencies
  4. Rollback One Service at a Time - Avoid rolling back multiple services simultaneously
  5. Verify After Each Rollback - Test thoroughly after each service rollback
  6. Monitor Closely - Watch system metrics and logs after rollback
  7. Plan Hotfixes - Have a plan to fix issues in the rolled-back version
  8. Consider API Contracts - Ensure API compatibility when rolling back services