robotstxt_app

Visual App for Testing URLs and User-agents blocked by robots.txt Files

Stars
4
View Code on GitHub Visit Website View on X
Ecosystems: Python

robots.txt Tester App

An App for testing URLs and User-agents blocked by robots.txt Files.

Given a certain robots.txt file, check whether or not any of the provided URLs can be fetched by all the user-agents in the robots.txt file (checks all the combinations).

Start testing!

Related Projects
crawler-user-agents

Syntactic patterns of HTTP user-agents used by bots / robots / crawlers / scrapers / spiders. pul...

07 Mar 2014 1,107
datasette-block-robots

Datasette plugin that blocks robots and crawlers using robots.txt

23 Jun 2020 3
RobotFramework

start automation test

26 Jun 2024 0
ChongLuaDao-Phishing-Blocklist

Machine-readable .txt blocklist of phishing URLs and IP Addresses from the Chống Lừa Đảo (https:/...

12 Aug 2023 2
UrlChecker

Detect the availability of the URLs with txt file or webpage 批量检测在线或本地文本中 URL 的有效性

14 Dec 2021 8
robotframework-anywherelibrary

AnywhereLibrary is a cross platform(desktop browser,Android,iOS) testing library for Robot Framew...

14 Apr 2014 66
webscraping-from-0-to-hero

The web scraping open project repository aims to share knowledge and experiences about web scrapi...

26 May 2022 1,533
PythonSpiderNotes

Python入门网络爬虫之精华版

19 Aug 2015 6,877
robotScraper

RobotScraper is a simple tool written in Python to check each of the paths found in the robots.tx...

16 Jun 2021 10
rpa_pricechecker

RobotFramework RPA web scraper automation in the cloud

14 Dec 2023 0
Robot-Framework

Test Scripts demonstrating different features of Robot Framework with Detailed Blog Articles. (ch...

27 Jun 2021 27
Ukrainian-EMA-blocklist

Machine-readable .txt blocklist of fraudulent URLs and IP Addresses from the Ukrainian Interbank ...

09 May 2023 2
GlobalAntiScamOrg-blocklist

Machine-readable .txt blocklist of scam URLs and IP Addresses from the Global Anti Scam Organizat...

21 Feb 2022 30