Skip to content

Latest commit

 

History

8 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 

Repository files navigation

Both the PHP and Python scripts are used to sync local mysql db to remote mysql db.

Features :

    1. It will check for the tables in the remote database,
          if the table not exists on the remote database,
          create the tables then feed the data from local.
  
    2. Subsequent calls will check the last_sync column and update data accordingly.
          if the table not exists on remote it will create it.
  
                      CREATE TABLE IF NOT EXISTS sync_status (
                            table_name VARCHAR(255) PRIMARY KEY,
                            last_sync TIMESTAMP NULL
                        )
  
   3. For very large tables, Batch Processing is used.
  
   4. No need to specify any table columns. 
         it will automatically get the structure using
         * SHOW Tables
         * DESCRIBE Tables
          

Requirements :

    1. You need to install the doctrine/dbal Package

        composer require doctrine/dbal
    
    2. All tables must have an Column that captures last_updated or modified time
              In my case I have time_stamp across all the tables.

PHP Limitations

 It took around 10 minutes to sync around 10k records due to it's limitations in parallel processing 

Python Record time SYNC

    1. Using Python's asyncio and aiomysql
        After creating all the tables
    2. Parallely we open a pool of mysql remote connections to all the tables
       Then for insertions we used batch process that operates in a async fashion.
       Achieved a record time for sync, around 1 lakh records.
        --- 10.03 Total Execution Seconds ---
        --- 0.17 Total Execution Minutes ---

!Screenshot 2024-08-27 014544 Screenshot 2024-08-27 015909 Screenshot 2024-08-27 014628

About

This PHP script is used to sync local mysql db to remote mysql db.

Resources

Stars

1 star

Watchers

1 watching

Forks

Releases

Packages

Contributors

Languages