Project Setup

1. Create project template by executing template.py file

2. Write the code on setup.py and pyproject.toml to import local packages.

3. Create a virtual env, activate it and install the requirements from requirements.txt

   i) Create a virtual environment named 'vehicle'

   python -m venv vehicle
   

   ii) Activate the environment (windows - cmd)

   vehicle\Scripts\activate.bat
   

   iii) To deactive the environment

   deactivate
   

   iii) Add required modules to requirements.txt, by doing

   pip install -r requirements.txt
   

4. Do a "pip list" on terminal to make sure you have local packages installed.

5. Include -e . in requirements.txt to install all the local packages from src folder in your "vehicle" environment.

6. Create a .env file and add the mongodb connection string.

CONNECTION_URL = <mongo-db_connection_string>

MongoDB Setup

1. Sign up to MongoDB Atlas and create a new project by just providing it a name then next next create.
2. From "Create a cluster" screen, hit "create", Select M0 service keeping other services as default, hit "create deployment" (NOTE : We store data inside clusters)
3. Setup the username and password and then create DB user.
4. Go to "network access" and add ip address - "0.0.0.0/0" so that we can access it from anywhere
5. Go back to project >> "Get Connection String" >> "Drivers" >> {Driver:Python, Version:3.12 or later}

   copy and save the connection string with you(replace <db_password>). >> Done.

6. Create folder "notebook" >> do step 7 >> create file "mongoDB_demo.ipynb" >> select kernel>python kernel>vehicle>>
7. Dataset added to notebook folder
8. Push your data to mongoDB database from your python notebook. We have to upload data in mongodb in key value format.
9. Go to mongoDB Atlas >> Database >> browse collection >> see your data in key value format

MongoDB Data Upload Heirarchy:

Organisation -> Project -> Cluster -> Database -> Collection

1. View Database

2. View Collections

3. Data Successfully uploaded to MongoDB Atlas Database

Logger and Exception Module Setup

Logged the exception

mlops-proj-vehicle-insurance-domain

Workflow for each component in data science lifecycle :

1. constant
2. config_entity
3. artifact_entity
4. component
5. pipeline
6. app.py / demo.py

DATA-INGESTION

1. Before working on "Data Ingestion" component >> declare variables within constants.init.py file
2. Add code to configuration.mongo_db_connection.py file and define the class for mongodb connection.
3. Inside "data access" folder , add code to proj1_data.py that will use monogo_db_connection.py. It will create a connection and fetches the from there.
4. To connect with database, fetch data in key-val format and transform that to a pandas dataframe.
5. Add code to entity.config_entity.py file till DataIngestionConfig class. NOTE : Set MONGODB_URL on command prompt using

set VAR_NAME=VALUE

To check if it is set ?

echo %MONGODB_URL%

Data Ingestion output

6. Add "artifact" directory to .gitignore file

Data Validation, Transformation, Model Trainer.

1. Complete the work on utils.main_utils.py and config.schema.yaml file (add entire info about dataset for data validation step)
2. Now work on the "Data Validation" component the way we did in step 17 for Data Ingestion. (Workflow mentioned below)
3. Now work on the "Data Transformation" component the way we did in above step. (add estimator.py to entity folder)
4. Now work on the "Model Trainer" component the way we did in above step. (add class to estimator.py in entity folder)

Data Validation Output

Data Transformation Output

Model Training Output

AWS SETUP

1. Login to AWS Console
2. Keep region set as us-east-1
3. Go to IAM -> Create new user (name it : firstproj)
4. Attach policy - give administrator access to access all aws functionalities
5. Go to user -> Security Credentials -> Access Keys -> Create access key
6. Select usecase as CLI -> agree to condition -> next -> Create access key -> Download csv file
7. Set env variables via CMD/Bash/Powershell

set AWS_ACCESS_KEY_ID=<access_key_here>
set AWS_SECRET_ACCESS_KEY=<secret_access_key_here>

IAM User created

8. Now add the access key, secret key, region name to constants.init.py
9. Add code to src.configuration.aws_connection.py file (To work with AWS S3 service)
10. Ensure below info in constants.init.py file:

            MODEL_EVALUATION_CHANGED_THRESHOLD_SCORE: float = 0.02
            MODEL_BUCKET_NAME = "<bucket_name_here>"
            MODEL_PUSHER_S3_KEY = "model-registry"

11. Go to S3 service >> Create bucket >> Region: us-east-1 >> General purpose >> Bucket Name: "<bucket_name_here>" >> uncheck: "Block all public access" and acknowledge >> Hit Create Bucket
12. Now inside "src.cloud_storage.aws_storage" code needs to be added for the configurations needed to pull and push model from AWS S3 bucket.
13. Inside src.entity dir we will have an "s3_estimator.py" file containing all the func to pull/push data from s3 bucket.

Model Evaluation Output

Model Pusher Output

1. n_estimators : 20 , rest same, no model in production initially.

2. n_estimators : 200, rest same, comparision happens now between the newly trained model and the model already present in production

newly trained model has f1 score higher than the model in production (i.e in s3 bucket), so the local newly trained model(the model inside the latest timestamp) is pushed to s3 bucket.