Project Overview
In this project, we crawl data from Foody website and ShopeeFood app. The dataset we crawl contains Restaurants, Comments, Dishes, Toppings. Then we model data for better visualization and analysis.
Details about Project
Crawl data from Foody
Frist we crawl data from Foody website using Requests.get() call API and the results are:
- A Restaurant table contains 101.401 rows and 33 columns, with each rows is a restaurant and each column is attribute of the restuarant(close to 130K restaurant in HCM at that time).
- A Comment table contains 319.378 rows and 64 columns, with each row is a comment.
Crawl data from ShopeeFood app
- Get API from ShopeeFood app using Virtual Android Machine and proxy.
- Because ShopeeFood app search for restaurant near your localtion, we draw a grid of longtitude and lattitude over HCM city using Nominatim, like:
- Crawl data by calling ShoppeeFood API, the results:
-
+ A Restaurant table contains 80.072 rows and 21 columns.
+ A Dish table contains 3.090.871 rows and 19 columns.
+ A Topping table contains 3.295.914 rows and 22 columns.
- Cleaning Data and EDA to understand data, write data into SQLite3.
Modeling Data
- Merge data from Foody and ShopeeFood app.
- Design and create table schema for data.
- Overall structure of the Database:
Visualization
Power BI DashBoard
- We build a DashBoard where you can search for the restaurant or the meal you want to eat. Or you can enter your location. The result are from the restaurant with 10km radius.
