w2-2-gw moving tool
The toolkit for moving IPFS wikipedia articles to the Great Web.
Status: alpha
data/links.csv
and don't remove it to avoid invalid transactions to networkgit clone https://github.com/SaveTheAles/wiki-crawler.git
cd wiki-crawler
pip3 install -r requirements.txt
config.py
with your personal credentials.data/queries.txt
with keywords you interested in for parsing. Every word from the new line.ifps daemon
python3 main.py
The crawler gets keywords from your data/queries.txt
and search for article titles on wikipedia by those keywords and create cyberlinks:
query -> [titles]
[titles] -> query
After that crawler gets every article in distributed wikipedia by the title it found and create cyberlinks:
[titles] -> [articles]
And finally, it gets links from the articles with query keyword and cyberlink them too:
[articles] -> [links]
All you created cyberlinks storing at data/links.csv
cids.py
- tool for extracting all CIDs you crawled to data/cids.txt
. Should be usefu if you need to pin your CIDs to the remote machine with IPFS node or IPFS cluster.
rpc_check.py
- tool for extra check if your address cyberlinked some cyberlinks. You can use it to avoid invalid transactions with already links existed.
wallet.py
and transaction.py
to cyber-py library and refactorrpc_check.py
as a parallel process