Client Background
Client: A leading eCommerce firm in the USA, Columbia, India, and Latin America
Gangala promotes local shops selling a wide variety of products at great prices. Easily find the best offers using our price comparison tool. It’s a WIN WIN for …
Industry Type: eCommerce
Services: e-commerce, retail business
Organization Size: 100+
Project Objective
To give User experience of easy and convenient Shopping by searching all the products like any medicines , Clothes , Gadgets etc in a single Website without going through all the E-Commerce Sites and make shopping easy and get the most affordable and best product.
Project Description
It’s an E-Commerce Sites that’s helps customer to compare different products that were available on different E-Commerce Sites like Flipkart , Amazon , Netmeds etc.It’s helps the user to visit only one sites to get what they need and find the perfect product without visiting all the sites.The gives the user a great and friendly Experience in Buying any Products.It’s Also have some Unique Similar Products Recommendation Based on user search and also have a ChatBot That’s solves User Query .It’s uses Big data and Rest API that’s help the projects for regular updates and regular fetching of the new products.
Our Solution
In BlackCoffer We create the flow of the Big Data and all Backend Solution That is requires for this futuristic E-Commerce Sites.We Create Pipelines for the data of all the products and their price and url fetch from different E-Commerce Sites using Custom made APIs And perform many data cleaning, data transformation and data validation techniques to make sure the standard of data to be used by Our Sites .We also get Additional Feature from the scraped data By using Different APIs . We also create automation and custom python scripts that helps us to achieve some outstanding data related tasks.
Project Deliverables
Python script for performing ETL and Cypher Query for big data Handling.
Tools used
- Jupyter Notebook
- DSS
- VS Code
Language/techniques used
- Python
- No SQl
- Cypher
- ETL
Models used
- Similar Price API
- Whatsapp Chat API
- Similarity Server to get similar products
Skills used
- Data Engineering
- Data Analysis
- Python Programming
- Rest APIs
Databases used
- DSS
- NEO4J
- MongoDB
Web Cloud Servers used
- Linode
- AWS
What are the technical Challenges Faced during Project Execution
- Data Cleaning : -The Scraped that will be used by our sites is coming from different sources and also it’s not’s that clean to be used by sites .This is the very first problem every data scientist faced during the whole process.
- Data Merging :- The data is scraped from around 140 sources that’s why it’s very difficult to maintain the attributes that should be used by all the sources and we can get a clean and sufficient amount of data to process.
- Data Validation :- There are many records that have null values and missing values that disturb the users experience a lot .That should be handle with very care.
How the Technical Challenges were Solved
- Data Cleaning : – For Data cleaning we used Python Data Frame and Pandas and data structure and handles the data cleaning and optimize our data for get correct data format and useful data.
- Data Merging : – For data Merging and data transformation we used pandas that help to get the appropriate data that can used further And also make Python pipelines for future updation.
- Data Validation :- For that data validation we use some fundamental property and feature selection that’s help us to make the appropriate data format and records to be used In our sites.
Project Snapshots
