A web scraper for helping individuals or companies to find out which kind of products are recommended more than others in specific category.
This is a fashion follower application for helping individuals or companies to find out which kind of products are recommended more than others in the category if the interest. The project is a web scraper and uses kimurai gem to fetch data from the Single Page Application. By default, the script is implemented brings the top 5 reviewed products from the specific category in the Trendyol.com. Users can also specify price ranges from the command line interface if needed. Both the category and the price range can be given from the command line interface. The category can be composed of several words, but it must be given in the Turkish language.
If the user enters
hp bilgisar
(computer means "bilgisayar" mean in Turkish) and 300-4000 as a price range, then it will search all computers with the hp brand that costs between 300 and 4000 Turkish Lira.
If the user enters
Nike siyah erkek ayakkabi
, it will search forblack Nike shoes for men
.
To get a local copy up and running follow these simple example steps.
You can run those functions in your own local environment. To run, you need to install RUBY on your computer. For windows, you can go to Ruby installer and for MAC and LINUX you can go to Ruby official site for instructions on how to install it.
Then you can clone the project by typing git clone https://github.com/RaminMammadzada/ruby-web-scraper.git
You must install the dependendies by bundling the Gemfile:
cd ruby-web-scraper
feature
branch by typing git checkout feature
gem install bundler
bundle update
bundle install
You must also have chromedriver
in your local development environment, otherwise you will get the error. It cannot be installed as a ruby gem.
On Mac OS:
brew
brew install chromedriver
curl http://chromedriver.storage.googleapis.com/2.38/chromedriver_mac64.zip -o chromedriver_mac64.zip
unzip chromedriver_mac64.zip
mv chromedriver /usr/local/bin
chmod +x /usr/local/bin/chromedriver
On Linux, and chromedriver (use proper path for your system and version):
wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | sudo apt-key add -
sh -c 'echo "deb http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list'
apt-get update
apt-get install google-chrome-stable
cd /tmp
wget https://chromedriver.storage.googleapis.com/2.44/chromedriver_linux64.zip
sudo unzip chromedriver_linux64.zip -d /usr/local/bin
rm -f chromedriver_linux64.zip
tmp
directory.cd ruby-web-scraper
.feature
branch by typing git checkout feature
../bin/main.rb
in the root file of the project. You can also type ruby bin/main.rb
in the root file of the project.This script was tested using RSpec which is a ruby testing tool. All public methods are tested.
gem install rspec
rspec --init
Open a terminal window and type rspec
All tests should be passed:
Ramin Mammadzada
Contributions, issues, and feature requests are welcome!
Feel free to check the issues page.
Give a if you like this project!
This project is MIT licensed.