PyFunceble
The tool to check the availability or syntax of domain, IP or URL.
APACHE-2.0 License
What has been fixed ?
-
We mitigated some cryptography vulnerabilities (CVE-2023-0286 & CVE-2023-23931) my changing the minimun required version.
-
A fatal error when no nameservers are configured by the hosting system. The default DNS Servers (when none are defined at OS/Configuration level) are now the Quad9 unsecured one. (#328)
-
The
git.ioservice is not deprecated. All URLs were Updated. (#341)
What has been improved?
For End-Users
-
The weebly.com SPECIAL rulesets. We now take down subdomains, that returns the 406 status code.
-
The wordpress.com SPECIAL rulesets. (#321)
For Developers
- We now support SQLAlchemy v2.x+.
- All SPECIAL rule handlers now share a set of methods to allow us to unify to way we go through our SPECIAL rulesets.
- All converters can now be used without initialization of the subject. Just use the
convertmethod, and you are good to go!
What has been removed?
- The support of any Python version lower that 3.8 (
<=3.7).
What is new?
For End-Users
-
The support of Python 3.11.
-
The support of PostgreSQL.
-
The "subject-switch" SPECIAL rulesets. (#185)
-
The "changeip" SPECIAL rulesets. (#311)
-
The imgur.com SPECIAL rulesets. (#319)
-
The eToxic SPECIAL rulesets. (#334)
-
An
ipsoutput subdirectory has been added. IPs are now "first-class citizens" among the plain text output format-s. (#268)
For Developers
- Testing with
pytestnow works out of the box. (#328)
Contributors
Thanks to those awesome people who contributed to this release!
- Josenilson Ferreira da SIlva - @Nilsonfsilva | @Nilsonfsilva
- smed79 - @smed79 | @smed79
- spirilln - @spirillen | @spirillen
- T145 - @T145 | @T145
Published by funilrys almost 2 years ago
๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐
๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐
Hello, World!
This is a maintenance release that provides some minor fixes regarding the handling of
several fatal errors.
Let the world know what you think or how you use PyFunceble with #PyFunceble or @PyFunceble on Twitter ๐!
I wish you all the best!
๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐
What was fixed ?
- A fatal error when an invalid YAML file is provided as a configuration file.
- A fatal error when a timeout is set to
0. - The requirements file for Windows users.
What was improved?
- The handling of errors produced by 2 of our dependencies.
- The handling of errors when using proxies.
What is new?
- The CLI help message now provide some basic examples.
- The CLI help message now provide some basic explanation about the argument "switch" mechanism.
Contributors
Thanks to those awesome people who contributed to this release!
- BigDargon - @bigdargon
- Imre Kristoffer Eilertsen - @DandelionSprout
- smed79 - @smed79
- spirillen - @spirillen
- T145 - @T145
Published by funilrys about 2 years ago
๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐
๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐
Hello, World!
This is maintenance release that provides some minor fixes regarding the handling of
non-UTF-8 characters and subjects without extensions (in some rare cases)
Let the world know what you think or how you use PyFunceble with #PyFunceble or @PyFunceble on Twitter ๐!
I wish you all the best!
๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐
What was fixed ?
- A fatal error when (in some rare cases) a subject without extension get through the requests handler.
- Issues regarding the handling of non UTF-8 characters.
Contributors
Thanks to those awesome people who contributed to this release!
- avatartw - @avatartw
- BigDargon - @bigdargon
- spirillen - @spirillen
Published by funilrys over 2 years ago
๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐
๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐
Hello, World!
This is a new release that provides a lot of new features, bug fixes and improvement.
In this release, the dr.ag and altervista.org domains will be checked against our set of SPECIAL rules. The syntax checker of PyFunceble was always following RFCs, but with this new release, it also accept some of the uncommon practices that can be found in the wild.
This release also bring registrar information among the provided datasets. Whether you use the Python API or the CLI, you can now read a registrar key or Registrar column that indicates where the domain is registrated. If you choose to actively use that feature, a TOP 10 list of registrar will be displayed at the end of your test sesssion.
Among other changes and new feature, the most important one may be the ability to instruct PyFunceble to use a global proxy or a specific one based on some set of rules that you can define within your configuration file. From now on, through this feature it is possible to test .onion alongside your normal test workflow - as example. You only have to instruct PyFunceble of the proxy it has to use and it will follow your order.
Let us know what you think or how you use PyFunceble with #PyFunceble or @PyFunceble on Twitter ๐!
I wish you all the best!
๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐
What was fixed ?
- A fatal error when a DNS name longer than 255 octets is given.
- The test of URL with scheme and port are no longer considered
INVALID. - All generated CSV file will be encoded in UTF-8.
- The version of
python-box.
What was improved?
- The syntax checker used to only follow RFCs. It now tries to accept and implement real world practices.
- The handling of IPv6 format when we give an IPv6 as DNS server.
- The WHOIS record decoded is now case insensitive. This will allow us to catch more that we use to.
- The check agians the autocontinue and inactive dataset is now done only when they are activated by the user.
- Our "self-healing" process that tries to avoid false-positive response.
- The deployment of AUR packages is now safer and quicker.
- The documentation.
What is new?
- The ability to use proxies when testing subjects.
- The ability to define proxy rulesets.
- The ability to get/read the domain registrar.
- The ability to get the TOP10 registrar - at the end of a test session.
- The
--max-http-retriesargument. - The
--*-lookup-onlyarguments. - The
--chancyargument. (WARNING: Use at your own risk.) - A URL test will check the status against the DNS server - first.
- The
dr.agSPECIAL rule. - The
altervista.orgSPECIAL rule. - A workaround against Y2K38 - until database vendors handle the problem.
- A way to automate CLI demos.
Contributors
Thanks to those awesome people who contributed to this release!
- avatartw - @avatartw
- Chris Griffith - @cdgriffith
- Mitchell Krog - @mitchellkrogza
- Fanboynz - @ryanbr
- spirillen - @spirillen
- Haris Guลกiฤ - @veracioux
Published by funilrys almost 3 years ago
๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐
Hello, World!
This is a new release that does some housekeeping and adds a new SPECIAL rule.
In this release, the fc2.com domain will be checked against our set of SPECIAL rules, the ascii encoding is better handled and some minor issue were fixed.
Let us know what you think or how you use PyFunceble with #PyFunceble or @PyFunceble on Twitter ๐!
I wish you all the best!
๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐
What was fixed ?
- A configuration upgrade issue.
- A DNS Resolver issue when null is decoded in incorrect URL.
What was improved?
- The handling of the
asciiencoding - around the CLI.
What is new?
- The
fc2.comSPECIAL rule.
Contributors
Thanks to those awesome people who contributed to this release!
- Imre Kristoffer Eilertsen - @DandelionSprout
- Mitchell Krog - @mitchellkrogza
- spirillen - @spirillen
- T145 - @T145
- ZeroDot1 - @ZeroDot1
Published by funilrys about 3 years ago
๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐
Hello, World!
I'm glad to announce this new major version of PyFunceble!
It took us some time, it it's time to release this new stable version.
This version provides a reinforcement of the Python module, deeper control of
the internals, and a rewrite of the internal data workflow.
The idea of this major version was mainly to reinforce the Python module to
make it usable "Out-Of-The-Box", but it went beyond my expectation with deeper
reviews, rewrite and discussion thanks to some people who chose to help by
using and giving feedback of what was one of the longest development period yet.
Therefore: Thank you to those who tested and contributed this new major
version from PyFunceble-dev 4.0.0a1 up to 4.0.0b67!
This version is the continuation of the adventure with some new features and
ideas.
A lot has been done, so you may have to discover by yourself through doing or
testing. But the most important changes are the support and expansion of CIDR
blocks, the ability to perform DNS over several protocols (UDP, TCP, HTTPS, TLS)
and the ability to safely configure PyFunceble over time through a new
configuration file called .PyFunceble.overwrite.yaml that will never be
updated by PyFunceble.
If you meet any new issue or behavior, you are invited to submit a new issue or
open a new discussion to seek help or assistance.
Let us know what you think or how you use PyFunceble with
#PyFunceble or
@PyFunceble on Twitter ๐!
I wish you all the best!
๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐
What was improved?
- The project structure.
- The source code.
- The documentation.
What is new?
- The ability to disable each of the internal test methods.
- The deletion of the "single" and "multiprocessing" modes.
- The ability to decode and test a whole IPv4 CIDR block.
- The ability to keep a safe and clean configuration file.
- ... and a lot more to discover!
Migration from 3.x
The migration to 4.x is normally done automatically through the configuration
checker - once you installed the new version.
A dataset migration will be triggered automatically. The migration may take
some time. Be ready to make a backup, cleanup your PyFunceble datasets or wait a
few minutes/hours.
In some rare cases, you may have to:
- delete your configuration file first.
- manually backup your local database tables and/or files.
- delete your local database tables and/or files.
Contributors
Thanks to those awesome people who contributed to this release!
- Imre Kristoffer Eilertsen - @DandelionSprout
- keczuppp - @keczuppp
- Mitchell Krog - @mitchellkrogza
- spirillen - @spirillen
- Yuki2718 - @Yuki2718
- Zachinquarantine - @Zachinquarantine
- ZeroDot1 - @ZeroDot1
Published by funilrys over 3 years ago
๐ ๐ ๐ ๐ ๐ ๐ ๐ ๐ ๐ ๐ ๐ ๐ ๐ ๐ ๐ ๐ ๐ ๐ ๐ ๐ ๐ ๐ ๐ ๐ ๐ ๐ ๐ ๐ ๐ ๐
Hello, World!
I hope that everything is going right for you and your loved one.
This release just provides some little fixes and improvements to the 3.3 version of PyFunceble.
It also announces the soon to be EOL of PyFunceble 3.x. Please find more about about the EOL here.
Let us know what you think or how you use PyFunceble with #PyFunceble or @PyFunceble on Twitter ๐!
I wish you all the best.
Stay safe and healthy!
๐ ๐ ๐ ๐ ๐ ๐ ๐ ๐ ๐ ๐ ๐ ๐ ๐ ๐ ๐ ๐ ๐ ๐ ๐ ๐ ๐ ๐ ๐ ๐ ๐ ๐ ๐ ๐ ๐ ๐
What is new?
- The EOL information and section in the documentation.
- The link to EOL information in the README.
- The usage of GitHub Action for the deployment to the PyPI and Docker Hub.
- The 2021 year in the LICENSE.
What was fixed?
-An issue regarding unhandled error while using multiple processes.
- The
python-boxrequirement declaration. - The
dnspythonrequirement declaration. - An issue regarding the missing configuration location when the platform could not be determined.
- The startup issue since the release of PyFunceble
4.0.0a1.
What was improved?
- The known issues section of the documentation.
- The RPZ and wildcard decoding.
- The way we send a transaction to the database.
- The way we handle errors when something was already delete from the database.
- The way we handle the auto continue under one of the supported CI engine.
- The way we handle the version comparison (backport).
Contributors
Thanks to those awesome people who contributed to this release!
- Avinash Reddy - @AvinashReddy3108
- Imre Kristoffer Eilertsen - @DandelionSprout
- spirillen - @spirillen
Published by funilrys about 4 years ago
๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐
Hello, World!
I hope that everything is going right for you and your loved one.
This release just provides a ๐ฅ fix of an issue regarding the migration from the old into the new layout.
Let us know what you think or how you use PyFunceble with #PyFunceble or @PyFunceble on Twitter ๐!
I wish you all the best.
Stay safe and healthy!
๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐
What was fixed?
- An issue regarding the migration of the automatic migration from the old into the new layout.
What was improved?
- The deployment of the stable version. We now reduce the deployment time by skipping a not necessary job.
What is coming ?
- Fix of reported issues.
- Improvement of the documentation (help welcome!).
- Introduction of possible and interesting features.
Contributors
Thanks to those awesome people who contributed to this release!
- spirillen - @spirillen
Published by funilrys about 4 years ago
๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐
Hello, World!
I hope that everything is going right for you and your loved one.
This release just provides some code maintenance along with a little improvement of the way we use SQLAlchemy during migration from old to the new layout.
Let us know what you think or how you use PyFunceble with #PyFunceble or @PyFunceble on Twitter ๐!
I wish you all the best.
Stay safe and healthy!
๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐
What was fixed?
- Warnings from dnspython.
- Warnings from domain2idna.
What was improved?
- The way we detect if the currently running package is the
devone. - The tests which relate to the syntax check.
- The tests which relate to the DNS resolver.
- The way we get the latest user agent.
- The presence of SQLAlchemy during migration from the old to the new layout.
What is new?
- A sanity check argument into
PyFunceble.load_configfor those who use the API.
What is coming?
- Fix of reported issues.
- Improvement of the documentation (help welcome!).
- Introduction of possible and interesting features.
Published by funilrys about 4 years ago
๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐
Hello, World!
I hope that everything is going right for you and your loved one.
This release just provides an improvement of the way we open and close a connection to a MySQL/MariaDB database.
Let me know what you think or how you use PyFunceble with #PyFunceble or @PyFunceble on Twitter ๐!
I wish you all the best.
Stay safe and healthy!
๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐
What was fixed ?
- An issue while saving user-inputted database credentials (if not found).
What was improved ?
- The way we open and close a connection to a MySQL/MariaDB database.
What is coming ?
- Fix of reported issues.
- Improvement of the documentation (help welcome!).
- Introduction of possible and interesting features.
Published by funilrys about 4 years ago
๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐
Hello, World!
I hope that everything is going right for you and your loved one.
This release just provides some improvements and a few new features.
There are a few important changes to notice. The first one is the switch from raw SQL to SQLAlchemy. The reason behind the switch is a change in the development process. I cannot endorse nor want to maintain raw SQL anymore.
To understand the matter, before the switch to SQLAlchemy I had code blocks that handled MySQL connections and statements and another one to handle MariaDB connections and statements. Most of the time it was 1 block for both but in other - not so peaceful - parts there were 2. Meaning that I was handling every use case by myself. With SQLAlchemy, we have 1 line/block for multiple database types. Of course we only support MariaDB and MySQL but it simplifies the integration of other database types in the future. The other reason which convinced me that it's the right way is that it forced me to review and simplify the table structure. Meaning that:
- We minimize the total size,
- We optimize the SQL flow,
- We minimize the number of SQL queries being made.
To simplify the switch and keep it as smooth as possible, a migrator is integrated into PyFunceble. That way, you don't have to take any special action, PyFunceble will handle everything.
The second important change is the sorting algorithms that were improved. Meaning that I simplified them to be efficient but before anything
correct. This change has been made because the sorting of digits was not human friendly. So I took advantage of the issue to completely improve the sorting algorithms.
The third important change is the introduction of new special rules which I let you discover through the documentation.
The fourth and last important change is the new --store-whois argument and logic. Before the introduction of this logic, we were storing the whois WHOIS record into the database. From now, it's only if it is needed by you. The idea behind this is to reduce the size of our database. Please note that the database migrator will follow this argument too.
There is a lot more but that's it for the most important changes!
Let me know what you think or how you use PyFunceble with #PyFunceble or @PyFunceble on Twitter ๐!
I wish you all the best.
Stay safe and healthy!
๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐
What was fixed ?
- Typos.
- Badges URL.
- An issue under macOS while using the
--multiprocessargument.
What was removed ?
- The
db_typesdirectory as it is now obsolete.
What was improved ?
- Some sections of the documentation.
- The standard sorting of file content.
- The hierarchical sorting of the file content.
- The cleanup logic behind the
--cleanand--clean-allarguments. - The way we process the mining of information.
- The way we open and close a database (MariaDB/MySQL) connection.
What is new ?
- The usage of SQLAlchemy instead of raw SQL.
- With an automatic migration routine if the old layout is detected.
- The
*.wix.comspecial rule. - The
*.hpg.com.brspecial rule. - The
*.000webhostapp.comspecial rule. - The
*.angelfire.comspecial rule. - Introduction of the
--store-whoisargument and logic. - A file position tracker which improve the continuation from an interrupted test.
- The mining capabilities are now available through the
--domainand--urlarguments. - The multiprocessing mode is automatically deactivated if less than 2 processes are allocated/allowed.
- The
--allargument prints everything. - We now retry 3 times with a sleep of 3 seconds between each while downloading vital files. If the URL is still not reachable after 3 retries, the system will stop.
- CI tests with Python 3.6.11, 3.7.8, and 3.8.5.
- The
PYFUNCEBLE_DEBUGandPYFUNCEBLE_DEBUG_ONSCREENenvironment variables.
What is coming ?
- Fix of reported issues.
- Improvement of the documentation (help welcome!).
- Introduction of possible and interesting features.
Contributors
Thanks to those awesome people who contributed to this release!
- spirillen - @spirillen
Published by funilrys over 4 years ago
๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐
Hello, World!
I hope that everything is going right for you and your loved one.
This release just provides a ๐ฅ issue which was provided by my last release.
Indeed, as I automate everything, I did not notice that the input file was being deleted.
The reason for the 3.2.2 instead of the 3.2.1 is because I did a mistake in the versioning a few days ago which resulted in me being forced to release 3.2.1 to PyPi and all other platforms.
Let me know what you think or how you use PyFunceble with #PyFunceble or @PyFunceble on Twitter ๐!
I wish you all the best.
Stay safe and healthy!
๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐
What was fixed ?
- An issue with the deletion of the input source at the end of a test.
What is coming ?
- Fix of reported issues.
- Improvement of the documentation (help welcome!).
- Introduction of possible and interesting features.
Contributors
Thanks to those awesome people who contributed to this release!
- spirillen - @spirillen
Published by funilrys over 4 years ago
๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐
Hello, World!
I hope that everything is going right for you and your loved one. This release just provides some fixes and very few new features.
This release includes one important change. Indeed, in the previous version, I included the notion of "shadow file" and made it default.
I now consider that decision as a bad one because it may have created unexpected behaviors especially while talking about execution time for some huge lists.
Therefore, from now, if you want to use that feature, you should use or append the --shadow argument or its configuration equivalent to access it.
The other important thing which is not a change is the deployment and provision of docker images and conda packages to the grand-public.
That means that from now, you can install PyFunceble (or its -dev counterpart) from PyPi, AUR, Docker Hub, and Anaconda Cloud. I spend a huge amount of time to automate the process. So, let me know if something is wrong with the two new repositories.
Let me know what you think or how you use PyFunceble with #PyFunceble or @PyFunceble on Twitter ๐!
I wish you all the best.
Stay safe and healthy!
๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐
What was fixed ?
- An issue with the creation of files when the
output/directory is missing. - An issue with some analytic files which were named
NoneNone. - An issue with the quiet mode which was not completely quiet while using the
-margument.
What was removed ?
- The direct dependency on
urllib3.
What was improved ?
- The documentation.
- The tests of the AdBlock decoder.
- The way we handle subdomains which are behind a public suffix.
- The way we generate the "shadow file". It can now take advantage of multiprocessing.
What is new ?
- Deployment to Docker Hub.
- Deployment to Anaconda Cloud.
- While testing domains, we convert them to lowercase.
- CI tests with Python 3.6.10, 3.7.7, and 3.8.3.
- A way to define the location of the
output/directory. It can be done through thePYFUNCEBLE_OUTPUT_LOCATIONenvironment variable. - The "shadow file" is not generated, nor used by default. You should explicitly give us permission to generate and use it through the
--shadowargument or its configuration counterpart.
What is coming ?
- Fix of reported issues.
- Improvement of the documentation (help welcome!).
- Introduction of possible and interesting features.
Contributors
Thanks to those awesome people who contributed to this release!
- NeolithEra - @NeolithEra
- spirillen - @spirillen
- ZeroDot1 - @jawz101
Published by funilrys over 4 years ago
๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐
Hello, World!
I hope that everything is going right for you and your loved one. This release took a bit of time because it was hard to find some time to give it some love. Anyway, here we are!
This release includes 2 important changes. The first one, is the fix of the way we connect to your MariaDB or MySQL database. There was a bug because I assumed that PyMySQL will close the connection by itself while using the with-statement. It turns out that it was not the case.
The second one and probably the most important one, is the inclusion of the self-cleaning of the inactive database. After collecting data in your local storage, this feature was highly wanted and here it is. Indeed, before today, the system was collecting and saving information into the inactive database but it never cleaned the unneeded one. From now, and after 28 days sitting in there, a subject will be automatically removed.
I spend most of my time restructuring the documentation so that we could export it into PDF without having to lose the structure. I was a bit shocked by the generated PDF, that I restructured the whole documentation for it to be rendered correctly. As you know, using PyFunceble requires an internet connection so PDF won't be a big deal. But on my side, I would like to be able (and to start) to download the PDF, print it, and correct what I wrote. That's how I will try to work in the coming month in order to complete the documentation.
By the way, I would like to thank all of those who contributed to the documentation! A lot of work is still waiting but we will come that far one day.
Let me know what you think or how you use PyFunceble with #PyFunceble on Twitter ๐!
I wish you all the best.
Stay safe and healthy!
๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐
What was fixed ?
- A MySQL/MariaDB connection and possibly communication issue.
- The multiprocessing logic.
- The length of the first line of all output files.
- The coloration of the PyFunceble ASCII-Art at the end which may not have been correct while using the MySQL or MariaDB database.
What was improved?
- The configuration loader (completely rewritten).
- The behavior behind the generation of the
.pyfunceble-envfile for those who use the MySQL or MariaDB database. - The help messages.
- The documentation's grammar, spelling, and structure.
- README's grammar and spelling.
- The structure of the inactive database (JSON).
What is new?
- A section in the documentation about some limitation which we may meet while using the MySQL or MariaDB database.
- The
--dotsargument and logic. - The self-cleaning of the inactive database after 28 days (configurable).
What is coming?
- Fix of reported issues.
- Improvement of the documentation (help welcome!).
Contributors
Thanks to those awesome people who contributed to this release!
- AnonymousPoster - @AnonymousPoster
- jawz101 - @jawz101
- Mitchell Krog - @mitchellkrogza
- spirillen - @spirillen
Published by funilrys over 4 years ago
๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐
Hello, World!
I'm glad to announce this new major version of PyFunceble!
It took me some time, but it's time to release this new version. This version provides a complete different tool. Indeed, the structure and the components of the project are totally different.
The idea of this major version was mainly to review the structure, but it went beyond my expectation thanks to the awesome users and contributors who gave some amazing feedback about PyFunceble(-dev).
One of the most fun I had, among the fix and development of new features, is the implementation of a total automation behind the generation and distribution of the IANA and Public Suffix databases file and the User-Agents. Indeed, before this version, I had to generate them and your own instance of PyFunceble had to fetch it after each release. From now, they are totally automated, and your own instance of PyFunceble will fetch and get the latest one when it judge that it's necessary to update your local version (generally after 24 hours).
This version is also the beginning of a new adventure, some new features and ideas.
Between what you may discover by yourself in this version, there is the support of GitLab CI/CD, IPv6 and even DNS over TCP to just name them.
As a list maintainer, this version provides you a switch which will checks the reputation of the subjects in your list with the help of the AlienVault's public reputation data. I highly invite you to check that as I'm thinking about the inclusion or improvement of reputation data in the future.
As a lot has been fixed, therefore, I'm not going to list them here, but if there are questions about the changes in details or a behavior which changed, please let me know on Keybase!
Let me know what you think or how you use PyFunceble with #PyFunceble on Twitter ๐!
I wish you all the best!
๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐
What was improved?
- The project structure and source code.
- The documentation.
- The help menu/message.
What is new?
- The support of GitLab CI/CD.
- The support of Python 3.8.
- A way to check the reputation of a given subject (domain and IP).
- Then
.onionTDL is notINVALIDanymore. - The support of the port when you give a custom DNS server.
- The possibility to give a MySQL/MariaDB Unix Socket file as hosts.
- The possibility to work from 2 branches (distribution and test branch) with the CI mode.
-
*.github.iowhich return404are returned asINACTIVE. - The possibility to fork the CLI behavior (file generation) from the API.
- A proper logging mechanism for better debugging.
- PyFunceble now initiate and configure the GIT repository for you under the CI mode.
- The tests where completely rewritten.
- You will get some warning and message from the CLI if your configuration are a bit extreme.
- The possibility to set a cooldown time between each test in order to reduce stress and pressure to DNS, WHOIS and target servers.
- We now look for CNAME, DNAME, A, and AAAA when the NS record is not given.
- Automation of a lot of administrative tasks that I had to do manually. Indeed, from now, the IANA, Public Suffix and even the User-Agent are automated in order for you to always get the latest one.
- The doubleclick SPECIAL rule is removed.
- The AdBlock decoder now decodes everything possible.
- ... and a lot more to discover!
What is coming?
- Fix of reported issues.
- Improvement of the behavior of the Inactive DB.
- Improvement of the documentation.
Migration from 2.x
The migration to 3.x is normally done automatically through the configuration checker.
In some rare case (and before submitting a new issue), you may delete your configuration files (.PyFunceble_production.yaml and .PyFunceble.yaml) first.
Contributors
Thanks to those awesome people who contributed to this release!
- Daniel - @dnmTX
- Imre Kristoffer Eilertsen - @DandelionSprout
- jawz101 - @jawz101
- Kowith Singkornkeeree - @kowith337
- Mitchell Krog - @mitchellkrogza
- opav - @opav
- ScriptTiger - @ScriptTiger
- sjhgvr - @sjhgvr
- spirillen - @spirillen
- ZeroDot1 - @ZeroDot1
Published by funilrys over 5 years ago
๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐
Hello, World!
Here is a quick fix of the --clean-all argument.
Indeed before this release when --clean-all was used along with the MySQL, SQLite or MariaDB database type, everything was literally cleaned.
From now, everything except the WHOIS database table is deleted. Why not the WHOIS database?
It simple, it's actually a static file which saves information about domains and is used every time you retest something with PyFunceble to
reduce test time.
Let me know what you think or how you use PyFunceble with #PyFunceble on Twitter ๐!
I wish you all the best!
๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐
What was improved?
- The
--clean-alllogic while using a database type different from JSON.
What is coming?
- Fix of reported issues.
- Review of the tests.
Contributors
Thanks to those awesome people who contributed to this release!
- Mitchell Krog - @mitchellkrogza
Published by funilrys over 5 years ago
๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐
Hello, World!
Here is a quick fix of the --local argument (and its configurable equivalent).
Indeed before this release in some rare cases, it is possible that an AttributeError might be raised.
For more accuracy, I renamed the source IANA to Syntax. That's it for this new release!
Let me know what you think or how you use PyFunceble with #PyFunceble on Twitter ๐!
I wish you all the best!
๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐
What was fixed?
- An
AttributeErrorwhich appear in some rare cases while using the--localargument.
What was improved?
- The accuracy of the source for
INVALIDoutput. It is notIANAbutSYNTAX.
What is coming?
- Fix of reported issues.
- Review of the tests.
Published by funilrys over 5 years ago
๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐
Hello, World!
I'm glad to announce this new major version of PyFunceble!
Many of you (if not everybody) wrote, talked and/or mentioned that PyFunceble is a great idea but it takes some times if not days to test lists.
Let me invite you to review your input with this new version which offers the usage of multiprocessing (as an option) while testing for files.
This new version includes many improvements, review and sometimes complete rewrite of most (if not all) subsystems. As we are all human, I strongly invite you to report any issue which you might meet or get with this new version. A lot of things have been taken into consideration to minimize issue but I'm and you should still stay aware that even if we are stable enough, bugs can emerge from nowhere.
Let me know what you think or how you use PyFunceble with #PyFunceble on Twitter ๐!
I wish you all the best!
๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐
What was fixed?
- The generation of the
percentage.txtfile which did not work when the quiet/silent mode was in use. - The Git command output which was not showing up while using PyFunceble under Travis CI.
What was improved?
- The project structure.
- The documentation structure.
- The documentation.
- The way we do a DNS Lookup.
- The way we determine, manage and return the status of a subject.
- The way we imported some entities.
- The way we share states and information across classes/subsystems.
- The way we generate/update
iana-domains-db.json - The Python API (Mostly backend).
- The way we load the configuration file.
- The way we clean what needs to be cleaned.
- The way we clean everything we generated.
- The way we handle some exceptions.
- The way the databases/class/subsystems are structured.
- The inactive database subsystem.
- The whois database subsystem.
- The auto-continue subsystem.
- The default user-agent.
-
iana-domains-db.json. -
public-suffix.json. - The CLI dispatcher.
What is new?
- The support of some custom DNS Server for DNS Lookup.
- The environment variable
PYFUNCEBLE_CONFIG_DIRwhich will replacePYFUNCEBLE_OUTPUT_DIRin the future. - The support of dotenv files for environment variable declaration.
- The possibility to make a DNS lookup through our Python API.
- The possibility to make a WHOIS lookup through our Python API.
- The support of Python 3.7.3.
- The usage of Codacy code review.
- The test of a file using multiprocessing (optional).
- Logs sharing is now disabled by default.
- The possibility to generate and fill a normal
output/directory structure from the API. - The possibility to use a different database type: SQLite, MySQL, MariaDB ou JSON.
- The possibility to test complements.
- Complements are for example
www.example.orgifexample.orgis given and vice-versa.
- Complements are for example
- We now ignore the well-known list of reserved IPv4 address.
- If you need to test such IPs please use the
--localargument from the CLI or its configuration equivalent.
- If you need to test such IPs please use the
- A safety check if a domain is syntactical
INVALID.- Indeed, we try to resolve to confirm it's invalidity.
- The percentage which shows up (if needed) in simple mode.
- The version number is now appended into all generated files.
- The execution time (when needed) is now appended to
percentage.txt. - The support of explicit port while testing for URL.
- The possibility to know the status of subject indexed into the inactive database.
- The separation of API, CLI Simple, CLI File, CLI Multiprocessing in the code for more clarity.
What is coming?
- Fix of reported issues.
- Review of the tests.
Contributors
Thanks to those awesome people who contributed to this release!
- Daniel - @dnmTX
- Mitchell Krog - @mitchellkrogza
- ZeroDot1 - @ZeroDot1
Published by funilrys over 5 years ago
๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐
Hello, World!
This release provides a minor but still important fix of the auto continue subsystem. Indeed, thanks to @dnmTX, I was able to find out that the auto continue subsystem was working wrongly. In fact, I was first filtering the whole list before trying to get the part we are going to test. Now I get the part we are going to test then process the filtering of all elements which are into the different databases.
Otherwise, because it was asked to me privately, we now write stdout with the output of the git push command when under Travis CI.
Finally, dear Arch Linux (and derivated) users, please note that from now, the package name changed. Indeed, it is now pyfunceble for the stable version and pyfunceble-dev for the development version.
I wish you all the best!
๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐
What was fixed?
- The autocontinue subsystem.
What was deleted or moved somewhere else?
- The counter indexes from the configuration file are now internal.
What was improved?
- The way we initiate the autosave subsystem.
- Indeed, we now ensure that the directory structure has been built before initiating this subsystem.
What is coming?
- Addition of features proposed by individuals or organizations via GitHub issue, GitHub comments, email, Keybase (Personal) or Keybase (PyFunceble Chat).
- Fix of reported issues.
- Improvement of the actual features.
- Review of the tests.
Contributors
Thanks to those awesome people who contributed to this release!
- dnmTX - @dnmTX
Published by funilrys over 5 years ago
๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐
Hello, World!
This release provides some minor fixes and reviews. Indeed, thanks to profiling I was able to reduce some internal noise by disabling submodules which were loaded and not needed.
Otherwise, I also provide a fix of the expiration date extraction for .com.pl and every whois server which provides the format: expiration date: 2019.03.07 00:09:28 for the expiration date.
Because, yaml.load() is now deprecated (cf), this patch fix the deprecation warning by using the yaml.safe_load() instead.
An update of the public-suffix.json is also in the box!
I wish you all the best!
๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐๐ ๐
What was fixed?
- An issue regarding the
auto_continuesubmodule which was trying to generate some files even if it was not requested by the end-user. - An issue regarding the extraction of the expiration date for
.com.pldomains and others who may have the same format.
What was improved?
- The way we initiate several submodules.
- The way we do avoid calling submodules which were not needed.
- The way we catch if a domain is a subdomain.
- The way we load the configuration file.
- The way we handle - as an imported module - an empty or
Noneinput. - The
public-suffix.jsonfile.
What is coming?
- Addition of features proposed by individuals or organizations via GitHub issue, GitHub comments, email, Keybase (Personal) or Keybase (PyFunceble Chat).
- Fix of reported issues.
- Improvement of the actual features.
- Review of the tests.
Contributors
Thanks to those awesome people who contributed to this release!
- hawkeye116477 - @hawkeye116477