changes to step 1 and 2 for data-profiler, data-charts and markdown-report examples

src/data-charts/README.md

@@ -1,27 +1,36 @@
-# Data charts in R
+# README - Data-charts
 
-## Introduction
+## Overview
 
-This worked example is designed to show how an R script can be dockerised and could be deployed as a federated compute task with some basic characteristics that could be adapted to other use cases:
+This worked example is designed to show how an R script can be wrapped up as a docker container. To see a workflow diagram of the steps taken below go to [Containerising a script as a federated compute task](https://github.com/federated-data-sharing/common-api/blob/master/doc/User_Guide_Containerising_Tasks.md#containerising-a-script-as-a-federated-compute-task).
 
-- use a common base R docker image
-- install some dependencies
-- use a single R script as the main computation
-- configure an environment similar to the remote federated node
 
 To build up an understanding of how to run the scripts, the same task can be run locally, then via local Docker.
 
-The script itself finds any CSV files in the input folder, reads them in via `readr`, identifies numerical fields and plots histograms of each one.
+The script itself [data-charts.R](./data-charts.R) finds any CSV files in the input folder, reads them in via `readr`, identifies numerical fields and plots histograms of each one.
+
+A summary of the histogram charts genertated can be seen below: 
+![image](https://user-images.githubusercontent.com/91956839/145797849-b8324690-1873-409a-a30d-107ee5db96b1.png)
+
 
 ## Pre-requisites
 
-R 3.6.1 or greater should be installed locally for the example to be run locally. The base Docker image will include a suitable version of R.
+- R 3.6.1 or greater should be installed locally for the example to be run locally. Depending on the operating system (Mac, Windows, Ubuntu, etc) you are using sreach for the relevent installation steps. 
+- ``Rscript`` which is a R interpreter used to execute R commands saved in a file with extesion ".R" will be needed locally. 
+- The base Docker image will include a suitable version of R.
+
+
+## Step 1: Run script locally on command line
+
+- Copy the repository on to your local machine. 
 
-## Step-by-step
+- Create an `input` and an `output` folder here. Put one or more CSV files in the `input` folder under the directory ```.../src/data-charts```.
 
-Create an `input` and an `output` folder here. Put one or more CSV files in the `input` folder.
+![image](https://user-images.githubusercontent.com/91956839/144869174-6c533f6f-8772-4174-ab3a-8bbfb3279132.png)
 
-Run the script directly, on the command line:
+- Put one or more CSV files in the `input` folder.
+
+- Run the script directly, on the command line:
 ```sh
 rm output/*
 
@@ -32,16 +41,17 @@ Rscript data-charts.R
 ```
 Look at the output files to see the charts produced.
 
-Look at the output files to see the statistical summaries.
+## Step 2: Run containerised script via docker commandline 
+
+- Build the docker image
 
-Then, build the docker image
 ```sh
-docker build . -t data-profiler
+docker build . -t data-charts
 ```
 
 > Depending on your docker set up you may need to run this command prefixed by `sudo`
 
-Then run the container on the same local file:
+- Then run the container on the same local file:
 ```sh
 rm output/*
 
@@ -53,6 +63,13 @@ docker run -it\
 
 > These commands are also provided as shell scripts
 
-Running this as as a federated data sharing task
+## Step 3: Run containerised script via federated data sharing task
+
+> TODO
+
+moved this section from overview (didn't want to delete it for now) could be deployed as a federated compute task with some basic characteristics that could be adapted to other use cases:
 
-> TODO
+- use a common base R docker image
+- install some dependencies
+- use a single R script as the main computation
+- configure an environment similar to the remote federated node

src/data-charts/data-charts.R

@@ -1,3 +1,5 @@
+if (!require("tidyverse")) install.packages("tidyverse")
+
 library(readr)
 library(dplyr)
 library(ggplot2)

src/data-profiler/README.md

@@ -2,15 +2,28 @@
 
 ## Overview
 
-This is a simple worked example of a summarisation task that can be wrapped up as a docker container. It is intended to work on any tabular data it finds in the input folder specified. This means it can be used for data explorations: it makes few assumptions on the source data other than it being valid CSV.
+This is a simple worked example of a summarisation task that can be wrapped up as a docker container. To see a workflow diagram of the steps taken below go to [Containerising a script as a federated compute task](https://github.com/federated-data-sharing/common-api/blob/master/doc/User_Guide_Containerising_Tasks.md#containerising-a-script-as-a-federated-compute-task).
 
-The script itself [data-profiler.py](./data-profiler.py) used the [pandas](https://pandas.pydata.org/) library to create a brief statistical profile of each field in the source data and write these to an output file.
+This example is intended to work on any tabular data it finds in the input folder specified. This means it can be used for data explorations: it makes few assumptions on the source data other than it being valid CSV. 
 
-## Step-by-step
+The script itself [data-profiler.py](./data-profiler.py) uses the [pandas](https://pandas.pydata.org/) library to create a brief statistical profile of each field in the source data and write these to an output file.
 
-Create an `input` and an `output` folder here. Put one or more CSV files in the `input` folder.
+The summary result from each csv file should have the following structure: 
+![image](https://user-images.githubusercontent.com/91956839/144863924-a45cf273-6b82-49f6-bde9-9d1068d11d46.png)
 
-Run the script directly, on the command line:
+
+
+## Step 1: Run script locally on command line
+
+- Copy the repository on to your local machine. 
+
+- Create an `input` and an `output` folder under the directory ```.../src/data-profiler```. 
+
+![image](https://user-images.githubusercontent.com/91956839/140361598-e4eb71b2-f058-457c-9066-93022acb5e48.png)
+
+- Put one or more CSV files in the `input` folder.
+
+- Run the script directly, on the command line:
 ```sh
 rm output/*
 
@@ -21,14 +34,17 @@ python data-profiler.py
 ```
 Look at the output files to see the statistical summaries.
 
-Then, build the docker image
+## Step 2: Run containerised script via docker commandline 
+
+- Build the docker image
+
 ```sh
 docker build . -t data-profiler
 ```
 
 > Depending on your docker set up you may need to run this command prefixed by `sudo`
 
-Then run the container on the same local file:
+- Then run the container on the same local file:
 ```sh
 rm output/*
 
@@ -37,11 +53,10 @@ docker run -it\
      --mount type=bind,source="`realpath $(pwd)/output`",target=/mnt/output\
      data-profiler:latest
 ```
+Look at the output files to see the statistical summaries.
 
 > These commands are also provided as shell scripts
 
-Running this as as a federated data sharing task
+## Step 3: Run containerised script via federated data sharing task
 
 > TODO
-
-

src/data-profiler/data-profiler.py

@@ -51,6 +51,9 @@
             if 'top' in all_summary.columns:
                 all_summary = all_summary.drop(columns=['top']) 
 
+            # count number of missing/NaN values in each column and add to all_summary
+            all_summary['null_count'] = df.isnull().sum(axis = 0)
+
             # write out to output_folder
             all_summary_path = f'{output_folder}/{basename}_summary.csv'
 

src/rmarkdown-report/README.md

@@ -0,0 +1,76 @@
+# README - rmarkdown-report
+
+## Overview
+
+This worked example is designed to show how an R script can be wrapped up as a docker container. To see a workflow diagram of the steps taken below go to [Containerising a script as a federated compute task](https://github.com/federated-data-sharing/common-api/blob/master/doc/User_Guide_Containerising_Tasks.md#containerising-a-script-as-a-federated-compute-task).
+
+
+To build up an understanding of how to run the scripts, the same task can be run locally, then via local Docker.
+
+The script itself [generate-report.R](./generate-report.R) redners the Rmarkdown file [report.Rmd](./report.Rmd) and creates an html output with all the histogram files for the numerical variables as seen in the [data-charts example](https://github.com/solmazeradat-aridhia/common-api-examples/tree/solmazeradat-aridhia-patch-1/src/data-charts).
+
+## Pre-requisites
+
+- R 3.6.1 or greater should be installed locally for the example to be run locally. Depending on the operating system (Mac, Windows, Ubuntu, etc) you are using sreach for the relevent installation steps. 
+- ``Rscript`` which is a R interpreter used to execute R commands saved in a file with extesion ".R" will be needed locally. 
+- The base Docker image will include a suitable version of R.
+- ``Pandoc`` package which can be installed by running
+```
+sudo apt-get install pandoc
+```
+
+
+## Step 1: Run script locally on command line
+
+- Copy the repository on to your local machine. 
+
+- Create an `input` and an `output` folder here. Put one or more CSV files in the `input` folder under the directory ```.../src/rmarkdown-report```.
+
+![image](https://user-images.githubusercontent.com/91956839/145202630-1c53bf8f-de6b-4c86-b5a7-fa5555d5e7e7.png)
+
+
+- Put one or more CSV files in the `input` folder.
+
+- Run the script directly, on the command line:
+```sh
+rm output/*
+
+export CA_INPUT_FOLDER=./input
+export CA_OUTPUT_FOLDER=./output
+
+Rscript generate-report.R
+```
+Look at the output files to see the charts produced.
+
+## Step 2: Run containerised script via docker commandline 
+
+- Build the docker image
+
+```sh
+docker build . -t generate-report
+```
+
+> Depending on your docker set up you may need to run this command prefixed by `sudo`
+
+- Then run the container on the same local file:
+```sh
+rm output/*
+
+docker run -it\
+     --mount type=bind,source="`realpath $(pwd)/input`",target=/mnt/input\
+     --mount type=bind,source="`realpath $(pwd)/output`",target=/mnt/output\
+     generate-report:latest
+```
+
+> These commands are also provided as shell scripts
+
+## Step 3: Run containerised script via federated data sharing task
+
+> TODO
+
+moved this section from overview (didn't want to delete it for now) could be deployed as a federated compute task with some basic characteristics that could be adapted to other use cases:
+
+- use a common base R docker image
+- install some dependencies
+- use a single R script as the main computation
+- configure an environment similar to the remote federated node