You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Both the PHP and Python scripts are used to sync local mysql db to remote mysql db.
Features :
1. It will check for the tables in the remote database,
if the table not exists on the remote database,
create the tables then feed the data from local.
2. Subsequent calls will check the last_sync column and update data accordingly.
if the table not exists on remote it will create it.
CREATE TABLE IF NOT EXISTS sync_status (
table_name VARCHAR(255) PRIMARY KEY,
last_sync TIMESTAMP NULL
)
3. For very large tables, Batch Processing is used.
4. No need to specify any table columns.
it will automatically get the structure using
* SHOW Tables
* DESCRIBE Tables
Requirements :
1. You need to install the doctrine/dbal Package
composer require doctrine/dbal
2. All tables must have an Column that captures last_updated or modified time
In my case I have time_stamp across all the tables.
PHP Limitations
It took around 10 minutes to sync around 10k records due to it's limitations in parallel processing
Python Record time SYNC
1. Using Python's asyncio and aiomysql
After creating all the tables
2. Parallely we open a pool of mysql remote connections to all the tables
Then for insertions we used batch process that operates in a async fashion.
Achieved a record time for sync, around 1 lakh records.
--- 10.03 Total Execution Seconds ---
--- 0.17 Total Execution Minutes ---
!
About
This PHP script is used to sync local mysql db to remote mysql db.