Project Overview

In this project, I used Airflow to automatically extract data daily from GithubArchive dataset on BigQuery, and store it into a Paritioned Table. Then use Google Data Studio to Visualize data.

Details about Project

- First I created a Ingestion-time Partitioned table in BigQuery to store GitHub Dailty Activities.
- The first Airflow Dag's duty is to write new Partition to GitHub Daily Activities table daily.

    Tasks flow:
taskflow1

    Tasks Result:
taskResult1

- The second Dag use to check if dag_runs of every single day in the previous month run successfully.

    Tasks flow:
taskflow2

    Tasks Result:
tasks_result2
- Then I plug the GitHub Daily Activities table into Google Data Studio.
- Use Custom Query and Date range parameters to create a reponsive table in Google Data Studio. - Build Github Activities DashBoard, the DashBoard can change the Date Range to get difference result. From that the DashBoard show summary of the difference Date Range.

    GitHub Activities in December 2022

github db1

    GitHub Activities in January 2023

github db1

    GitHub Activities in December 2022 and January 2023

github db1

More Detail about this Project on GitHub