Skip to main content

Rollback Overview

This section provides comprehensive guides for rolling back deployments when issues are discovered in production or staging environments. Rollback procedures are critical for maintaining system stability and quickly recovering from problematic deployments.

Overview​

The SBM CRM Platform consists of multiple components, each with specific rollback procedures:

  • WeChat Mini Program - Rollback through WeChat Official Account Platform version management
  • Admin Panel - Rollback via GitHub Actions production pipeline
  • API Services - Rollback via GitHub Actions with service-specific selection

Each component has different rollback mechanisms and timeframes that must be understood before attempting a rollback.

When to Perform a Rollback​

Consider rolling back when:

  • Critical Bugs: Production issues that impact core functionality
  • Performance Degradation: Significant performance issues introduced by the new version
  • Security Vulnerabilities: Security issues discovered in the deployed version
  • Data Integrity Issues: Problems affecting data consistency or integrity
  • Service Unavailability: Services become unavailable or unstable
  • User Impact: Issues affecting a significant number of users
warning

Rollback should be a last resort after attempting to fix issues with a hotfix. Only rollback if a hotfix cannot resolve the issue quickly.

Rollback Workflow​

The general rollback workflow follows these steps:

1. Identify the Issue​

  • Confirm the problem is related to the recent deployment
  • Document the specific issues and their impact
  • Verify which version introduced the problem

2. Determine Target Version​

  • Identify the last known stable version
  • Verify the target version exists and is available
  • Check for any dependencies or compatibility requirements

3. Execute Rollback​

  • Follow the component-specific rollback procedure
  • Monitor the rollback process
  • Verify the rollback completed successfully

4. Post-Rollback Verification​

  • Test critical functionality
  • Monitor system health and metrics
  • Verify services are running correctly
  • Check application logs for errors

5. Document and Communicate​

  • Document the rollback reason and process
  • Communicate with the team about the rollback
  • Plan for addressing the issues in the rolled-back version

Rollback Guides​

WeChat Mini Program​

Rollback the WeChat Mini Program to a previous version through the WeChat Official Account Platform.

WeChat Review Time

WeChat rollbacks require review and approval, which can take 1-7 business days. Plan accordingly for urgent rollbacks.

Admin Panel​

Rollback the Admin Panel to a previous version via GitHub Actions.

API Services​

Rollback individual API services to previous versions via GitHub Actions.

  • API Rollback Guide - Complete guide for rolling back API services, including service selection

Prerequisites​

Before performing a rollback, ensure you have:

  • Access to the repository and GitHub Actions (for Admin Panel and API)
  • Access to WeChat Official Account Platform (for WeChat)
  • Appropriate permissions for the target environment
  • Knowledge of the version to rollback to
  • Understanding of the impact of the rollback

Best Practices​

  1. Maintain Version History - Keep detailed records of all deployments and their versions
  2. Test Rollback Procedures - Periodically test rollback procedures in staging
  3. Document Rollbacks - Always document why a rollback was performed
  4. Communicate Early - Notify the team before performing a rollback
  5. Verify After Rollback - Thoroughly test the system after rollback
  6. Plan Forward - Have a plan to address the issues that caused the rollback
  7. Monitor Closely - Watch system metrics and logs after rollback

Rollback Considerations​

Data Compatibility​

  • Ensure the rollback version is compatible with the current database schema
  • Check for any data migrations that may need to be reversed
  • Verify API contracts are compatible

Service Dependencies​

  • Consider dependencies between services when rolling back
  • Ensure rolled-back services are compatible with other services
  • Check for breaking changes in API contracts

User Impact​

  • Minimize user disruption during rollback
  • Communicate maintenance windows if needed
  • Plan for potential service interruptions

Troubleshooting Rollback Issues​

If you encounter issues during rollback:

  1. Check Version Availability - Verify the target version exists in the registry/repository
  2. Review Rollback Logs - Check GitHub Actions or platform logs for errors
  3. Verify Permissions - Ensure you have necessary permissions for rollback
  4. Check Service Status - Verify services are in a state that allows rollback
  5. Contact DevOps Team - Reach out for assistance with infrastructure issues