NetDiscovery

(DisruptorRedisKafkaRabitMQRocketMQ)(Http ClientOkhttp3Selenium)
RxJava 2
ParserPipeline
Request Queue
Kotlin
JS
Request header
debug Request debug Local Cache(RxCache)
PipelineRequestDownloadDomain
User Agent Cookies
Pipeline
URL
Spider SpiderEngine (EtcdZookeeper)
agent CPU
SpiderEngine Quartz
SpiderEngine httpRPC

|| ---|:-------------: netdiscovery-core-core| netdiscovery-core-engine| netdiscovery-core-engine-monitor| netdiscovery-downloader-htmlunit| netdiscovery-downloader-httpclient| netdiscovery-downloader-okhttp| netdiscovery-downloader-selenium| netdiscovery-pipeline-couchbase| netdiscovery-pipeline-elasticsearch| netdiscovery-pipeline-mongo| netdiscovery-pipeline-redis| netdiscovery-queue-kafka| netdiscovery-queue-rabbitmq| netdiscovery-queue-redis| netdiscovery-queue-rocketmq| netdiscovery-kotlin-coroutines| netdiscovery-kotlin-dsl|

NetDiscovery Vert.xRxJava 2

Java gradle jcenter() module build.gradle

Gradle

repositories {
    jcenter()
}

Maven

<repositories>
    <repository>
        <id>central</id>
        <name>bintray</name>
        <url>https://jcenter.bintray.com</url>
    </repository>
</repositories>

NetDiscovery Spider Spider SpiderEngine

SpiderEngine Etcd/Zookeeper monitor SpiderEngine

Stargazers over time

wiki

:

user-agent-list:user agent
FXHParser:

TODO List:

Kotlin Kotlin-Cli
OpenCV Tesseract OCR
admin
Consul
HtmlUnit
chromium
wiki
RxJava 3.x
Java 11
docker

Contributors:

Contributing:

Pull requests are welcome

:

Wechatfengzhizi715

JavaAndroid

License

Copyright (C) 2017 - present, Tony Shen.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

   http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

Badges

Extracted from project README