bigData

First create an environnement :

Run this code for a mock run on a sample of 0.1% of the data :

Note: the results are insignificant on sampled data.

Steps to get the resulting clusters from our analysis in the juypter notebook :

Go to drive : https://drive.google.com/drive/folders/1GOjfOQzDX0sPl-huZJgGX2NFV5F1rGdw?dmr=1&ec=wgc-drive-globalnav-goto&hl=fr
The link is on google drive, since the data exceeds the 2 GB limit for other websites
And put the data in the data_clean folder
Go into the jupyter notebook : covariance_matrix.ipynb
Run the differents cells and choose if you just want to run the demo with the boolean argument in the main() definition

The whole data from the google drive must be saved in processed/final_yearly

To run the code on the whole data :

Note: running the code on the whole data takes multiple hours (3+).

Name		Name	Last commit message	Last commit date
Latest commit History 35 Commits
processed/final_yearly_sample		processed/final_yearly_sample
Financial_Big_Data_report.pdf		Financial_Big_Data_report.pdf
README.md		README.md
clustering_script.py		clustering_script.py
covariance_matrix.ipynb		covariance_matrix.ipynb
discussion.ipynb		discussion.ipynb
environment.yml		environment.yml
flash_crash_detection.ipynb		flash_crash_detection.ipynb
flash_crash_detection.py		flash_crash_detection.py
main_subset.py		main_subset.py
preprocessing.ipynb		preprocessing.ipynb
preprocessing.py		preprocessing.py
requirements.txt		requirements.txt

Navigation Menu