Skipping v2.5.2 because I accidentally released v2.5.1 as internal version "v2.5.2-dev".

• Fixed Twitter Scraper - previously used the now archived https://github.com/n0madic/twitter-scraper
  • Updated to fork maintained by imperatrona - https://github.com/imperatrona/twitter-scraper
• sendFile data keys were previously very limited, now it supports the same data keys as discord messages / downloads.
• Data key {{serverName}} is more reliable, previously often failed to parse. Discord-related data keys are now more reliable in general.
• Added subfoldersFallback setting alongside subfolders. If this setting is present when subfolders encounters an unparsed data key ({{example}}), it will use subfoldersFallback in place of subfolders.
• Fixed line breaks not parsing correctly for sendFileCaption
• Added data key {{usernameNoLeadPeriod}} - username of message author but with leading periods (".") stripped.
• 403 will once again cause a download to immediately bail. A recent update undid this and made it retry X times.

v2.5.1 Patch:

• Fixes numerous longtime issues with user accounts, particularly regarding thread functionality and history processing.

v2.5.0 (released earlier today):

• Fixed Discord CDN 403 errors.
• Restructured how the bot handles sources (servers/categories/channels/threads).
  • Seems to work much better for forums and threads. Note that user accounts cannot access active threads, only archived.
  • Issues may occur, report anything related to this.
• Fixed issues with history date range filtering.
• Added commandPrefix to source configs.
  • Override the command prefix per-group, including leading space, (default "ddg ").
• blockedReactions & allowedReactions in filters now accepts standard Discord emoji names.
• Added support for Instagram 2FA. Use instagramTOTP in the credentials section with your TOTP seed, or enter your OTP code upon login. TOTP seed should always work, OTP code will apply to your login cache but if the cache expires, you'll need to re-enter the code upon new login.
• Added filepathNormalizeText & filepathStripSymbols to global config, source configs, and source message/download logging.
  • Both false by default.
  • Normalize text will convert letter-like symbols/fonts to their standard equivalent.
  • Improved filepath clearing padded spaces. Generally improved filename/filepath scrubbing.
• Improved reliability of fetching server and channel data.
• Added outputHistoryStatus & outputHistoryErrors globally and per-source to optionally disable history outputting.
• Added content setting to download & message logging. Currently accepts {{message}} for message logging and {{msgContent}} and {{link}} and {{embedDesc}} for download logging.
• Added {{attachmentID}} key to filenameFormat data keys.
• Changed subfolder data key {{userName}} to {{username}} for consistency.
• Fixed nil error crash when encountering command-message error while handling history.
• Restructured various areas, issues could occur, open an issue if you encounter anything.
• Upgraded Dependencies

• Fixed Discord CDN 403 errors.
• Restructured how the bot handles sources (servers/categories/channels/threads).
  • Seems to work much better for forums and threads. Note that user accounts cannot access active threads, only archived.
  • Issues may occur, report anything related to this.
• Fixed issues with history date range filtering.
• Added commandPrefix to global config and source configs.
  • Override the command prefix per-group, including leading space, (default "ddg ").
• blockedReactions & allowedReactions in filters now accepts standard Discord emoji names.
• Added support for Instagram 2FA. Use instagramTOTP in the credentials section with your TOTP seed, or enter your OTP code upon login. TOTP seed should always work, OTP code will apply to your login cache but if the cache expires, you'll need to re-enter the code upon new login.
• Added filepathNormalizeText & filepathStripSymbols to global config, source configs, and source message/download logging.
  • Both false by default.
  • Normalize text will convert letter-like symbols/fonts to their standard equivalent.
  • Improved filepath clearing padded spaces. Generally improved filename/filepath scrubbing.
• Improved reliability of fetching server and channel data.
• Added outputHistoryStatus & outputHistoryErrors globally and per-source to optionally disable history outputting.
• Added content setting to download & message logging. Currently accepts {{message}} for message logging and {{msgContent}} and {{link}} and {{embedDesc}} for download logging.
• Added {{attachmentID}} key to filenameFormat data keys.
• Changed subfolder data key {{userName}} to {{username}} for consistency.
• Fixed nil error crash when encountering command-message error while handling history.
• Restructured various areas, issues could occur, open an issue if you encounter anything.
• Upgraded Dependencies

• Disabled immediate cancellation when encountering 403, retries like other errors (except 404), seems to fix recent Discord CDN errors.
• Restructured how the bot handles sources (servers/categories/channels/threads).
  • Seems to work much better for forums and threads. Note that user accounts cannot access active threads, only archived.
  • Issues may occur, report anything related to this.
• Fixed issues with history date range filtering.
• Added commandPrefix to global config and source configs.
  • Override the command prefix per-group, including leading space, (default "ddg ").
• blockedReactions & allowedReactions in filters now accepts standard Discord emoji names.
• Added support for Instagram 2FA. Use instagramTOTP in the credentials section with your TOTP seed, or enter your OTP code upon login. TOTP seed should always work, OTP code will apply to your login cache but if the cache expires, you'll need to re-enter the code upon new login.
• Added filepathNormalizeText & filepathStripSymbols to global config, source configs, and source message/download logging.
  • Both false by default.
  • Normalize text will convert letter-like symbols/fonts to their standard equivalent.
  • Improved filepath clearing padded spaces. Generally improved filename/filepath scrubbing.
• Improved reliability of fetching server and channel data.
• Added outputHistoryStatus & outputHistoryErrors globally and per-source to optionally disable history outputting.
• Added content setting to download & message logging. Currently accepts {{message}} for message logging and {{msgContent}} and {{link}} and {{embedDesc}} for download logging.
• Added {{attachmentID}} key to filenameFormat data keys.
• Changed subfolder data key {{userName}} to {{username}} for consistency.
• Fixed nil error crash when encountering command-message error while handling history.
• Restructured various areas, issues could occur, open an issue if you encounter anything.
• Upgraded Dependencies

!! v2.3.0 was released last week, see changelog for settings breaking changes !!

• Fixed critical issues with the new "subfolders" setting.
• Removed "#" from {{channelName}} for "subfolders".
• Added "alias" (string) and "aliases" (list of strings) to sources. These keywords can be used to call history commands. If a server/channel/etc config has something like "alias": "abc", you can use ddg history abc. Multiple configs can use the same aliases to call them all with 1 command. Aliases will not work if they contain spaces or commas.
• logLinks and logMessages reworked - "divideBy____" settings removed and replaced with "subfolders" and "filenameFormat" like the download sources.
• logLinks and logMessages global default options added (unused by default).
• Improved Discord source validation at startup, added source Discord permission checking to source validation.
• Improved a lot of logging appearance and info.
• Some log outputs formerly sent under "debug" are now under "verbose".
• Upgraded Dependencies

logLinks & logMessages Data Keys

Applies to subfolders, filenameFormat, linePrefix, lineSuffix WITHIN logLinks and logMessages, globally and per-source.

!! IMPORTANT !!
All divideBy\_\_\_\_\_ settings have been removed and been replaced with subfolders, see below.

• Added "subfolders" and removed all divideBy settings, a list of strings for each subfolder title.

  • Replaceable keys are: {{serverID}}, {{serverName}}, {{categoryID}}, {{categoryName}}, {{channelID}}, {{channelName}}, {{userID}}, {{userName}}, {{fileType}}, {{year}}, {{monthNum}}, {{dayOfMonth}}, {{hour}}, {{message}}, {{messageID}}.
  • Default is "subfolders": [ "{{fileType}}" ],.
  • Present both globally in the main settings and individually in source settings. If these are missing from source (i.e. no override), it will use the main settings as default.
  • Something like "subfolders": [ "{{serverName}}", "{{categoryName}}", "{{channelName}}", "{{year}}", "{{fileType}}" ], will equate to "BASE_DESTINATION/My Server/Category/Channel/2023/images/".

• Reworked Emoji & Sticker Support

  • Removed ddg emojis command.
  • Runs at launch after all other initial startup tasks finish.
  • Added settings to main settings group:
    • "emojisServers", "stickersServers" are lists of server IDs to download from. ex: "emojisServers": [ "123", "456" ],
    • "emojisFilenameFormat", "stickersFilenameFormat" to override the filename format. Default is "{{ID}} {{name}}, only accepts {{ID}} and {{name}} as keys.
    • "emojisDestination", "stickersDestination" to override the destination folders. If these settings are missing, it will use "emojis" and "stickers" within the bot folder. The bot will automatically create subfolders by channel name within whatever destination.

• Hopefully fixed Discord timestamp snowflake issues regarding history date range filters.

• Fixed error upon exit if Twitter / X is disabled.

• Added "verbose" and "debugV" but they currently do nothing, just future variants of "debug".

• Fixed history command not finding channels when going by category.
• Fixed history not working on news / announcement channels.
• Fixed history command including categories themselves in history queues when going by server or by all.
• Added warning to history command if the bot detects issues with Discord cache.
• Fixed filters.allowedFilenames not working.
• Fixed some minor crashing issues.
• Optimized and cleaned up some things.

• Fixed folder division sometimes causing fatal crash.

• Settings now supported in .YAML format. Settings are identical to the JSON structuring, just an alternative format option. Upon launch, if using a .JSON/.JSONC config, the bot will translate it to "cache/settings.yaml" and alternatively when using a .YAML config it will translate it to "cache/settings.json".
• Reduced which settings are hidden in settingsOutput and program-generated settings.
• Twitter authentication now caches to reduce login requests.
• Fixed "x.com" links not being recognized as Twitter.
• If not authenticated with Twitter/X, those links are skipped entirely to avoid hangups.
• Download speed metrics are far more accurate, factoring the elapsed file download duration rather than overall duration.
• Default for downloadRetryMax changed from 3 to 2.
• Added setting to main section: overwriteDefaultReaction (text), "✅" by default.
• Added setting to main section: overwriteCachePath (text), "cache" by default.
• Added setting to main section: overwriteHistoryPath (text), "cache/history" by default.
• Added setting to main section: overwriteDuploPath (text), "cache/.duplo" by default.
• Added setting to main section: overwriteTwitterPath (text), "cache/twitter.json" by default.
• Added setting to main section: overwriteInstagramPath (text), "cache/instagram.json" by default.
• Added setting to main section: overwriteConstantsPath (text), "cache/constants.json" by default.
• Added setting to main section: overwriteDatabasePath (text), "database" by default.
• Added setting to main section: overwriteDatabaseBackupsPath (text), "backups" by default.
• Added setting to main section: historyManagerRate (number), 5 by default, seconds between rechecking the queued history jobs.
• Added setting to credentials section: twitterEnabled (bool), true by default, optionally completely disable the Twitter Scraper section.
• Added setting to credentials section: instagramEnabled (bool), true by default, optionally completely disable the Instagram Scraper section.
• Added setting to credentials section: twitterProxy (text), if this setting isn't blank, the Twitter scraper will use it as a proxy address.
• Added setting to credentials section: instagramProxy (text), if this setting isn't blank, the Instagram scraper will use it as a proxy address.
• Added setting to credentials section: instagramProxyInsecure (bool), false by default.
• Added setting to credentials section: instagramProxyForceHTTP2 (bool), false by default.
• Added setting to main section: logOutput (text), if this setting isn't blank the program will log all console output as a text file, the setting should be a complete filepath including .txt or the desired extension.
• Added setting to main section: logIndent (bool), enabled by default, if disabled the program will not indent between timestamps and log content.
• Fixed logMessages and logLinks timezones.
• Improved Twitter & Instagram authentication logging.
• Improved general output.
• Improved History status output.
• Optimized some things.
• Upgraded dependencies.
• Fixed some typos.

Over the next week (since posting this) I'll be improving the readme.

• Fixed Twitter Scraper (I didn't actually do anything, the developer updated it).
• Added commandTagging to main settings. True by default. Controls whether command senders should be tagged in replies from the bot.
• Added historyRequestDelay to delay between requests of history messages.
• Added historyRequestCount to limit messages per-request in history. Default and max value is 100.
• Added delayHandling and delayHandlingHistory (in milliseconds) to delay handling/downloading.
• Added divideByDay and divideByHour to main settings and source settings.
• Fixed categoryBlacklist not functioning properly.
• Fixed blockedPhrases and allowedPhrases not ignoring empty entries.
• Download errors no longer tag sender
• savePossibleDuplicates now false by default
• Updated Websocket Dependency for security patch
• Updated Dependencies

• Twitter - They ditched their Free API tiers and banned access to all of those existing applications. I've replaced it with a Twitter Scraper library. It's more crude than the official API was but at least it works. I've noticed it will only download the first video/gif if there's multiple in 1 tweet. It seems to be fine for images. I'll see if I can fix this soon. Every other aspect of it seems to be fine so far. You can optionally login via email+password ("twitterEmail" and "twitterPassword" in the credentials section) to download from private accounts your user follows.
• Thread Support - proper thread support, even for users (mostly). Treats threads as subchannels of a channel, you can use direct thread IDs or the forum channel ID. It will fetch any archived or active threads it can find. USER ACCOUNTS CANNOT FETCH ACTIVE THREADS, ONLY ARCHIVED. Message logging works properly for threads.
• Improved general reliability of how the bot checks channel and server data.
• Fixed unwarranted Discord permission errors, particularly regarding user accounts and DM channels.
• Fixed unwarranted Discord source invalidation warnings, particularly regarding DM channels.
• Fixed categories not being handled properly by history.
• Added "watchSettings" (true or false) to main settings. True by default. Controls whether settings are updated live when the file is modified.
• Added "backupDatabaseOnStart" (true or false) to main settings. False by default. Backs up database folder into a dated zip file under a local "backups" folder. This feature can be expanded upon request.

SEE PRIOR RELEASE NOTES FOR v1.X.X TO v2.0.0 CHANGES

• Fixed History Before/Since Filtering not working or partially working
• Improved History Caching (for when it's cancelled, interrupted, crashes, etc)
• Updated Dependencies

NOT ALL CHANGES MAY BE LISTED HERE, I TWEAKED A LOT

SEE SETTINGS CHANGELOG BELOW

I'm going to stop using the automated builds for releases and post all builds that succeeded. Let me know if any don't work for certain architectures.

• HISTORY

  • Job queue system for better maintaining and monitoring history processes. Runs jobs in order now.
  • Runs jobs in limited concurrency, specify # of jobs to run at a time with historyMaxJobs, limited by processLimit.
  • Reworked history command structure and fixed too many things to list.
  • Added history list (or status or output) subcommand for outputting job statuses.
  • Fixed date range not accounting for local timezones.
  • Redundancy to ensure it fetches messages to true start of channel.
  • Added history dbwipe/wipedb subcommand to purge the specified channel(s) from the download database.
  • Added history cachewipe/wipecache subcommand to purge the specified channel(s) from the cache folder.

• UPGRADED DISCORDGO & Fixed Related Issues - Now using the latest release of discordgo, I modified it to re-allow user accounts to be authenticated.

  • Fixed the int/float conversion error many people encountered.

• Added Instagram Support - requires account username & password. Sometimes requires captcha popup completion.

• Added Thread Support (for genuine Discord Bots ONLY)

• Added Users as Sources - download/monitor from any channel the bot can access so long as the message belongs to specified users. Settings follow channels.

• Added Categories as Sources - Settings follow channels.

• Removed Google Drive support, will re-add in the future, the prior one was ported from Seklfreak's original.

• Upgraded all dependencies.

• Removed Mastodon support

• Fixed Reddit support (if building yourself, GODEBUG=http2client=0 before build)

• Discord, Twitter, Instagram logins retry 3-5 times if failed.

• Fixed issue with Twitter API ignoring certain urls.

• Fixed several nil crashes.

• Commands run asynchronously now.

• Changed a lot of console output styling and formatting.

• Optionally target specific settings file to run multiple instances off one executable & database with arg1, no file extension, example: ddg.exe customSettingsFile

• Added Filesize info for downloads

  • Crude download speed metrics

• Better handles download errors. Retries X times unless 404 or 403.

• Removed permissions check, won't check if user account since it always errors.

• Added checkup / heartbeat console output, control rate with checkupRate

• Admin channel status messages alert to detected settings modifications.

• filenameFormat now also uses generic data key replacements from presence.

SETTINGS CHANGES:

I changed a lot of things, see the examples in the README for the full list and comments.

• (credentials) Removed userBot. Automatically detects and corrects if needed
• (credentials) Added instagramUsername, instagramPassword
• (admin channels) Added logProgram
• (main) Added users, categories
• (main) Added settingsOutput
• (main) Added messageOutputHistory
• (main) Added processLimit, historyMaxJobs
• (main) Added checkupRate, connectionCheckRate, presenceRefreshRate
• (main) Added sendAutoHistoryStatus
• (main) Added allBlacklistUsers, allBlacklistCategories
• (main) Added allowCommands to set default for all sources
• (main) Added divideByChannel to set default for all sources
• (main) Added divideByMonth to set default for all sources
• (main) Added divideByServer to set default for all sources
• (main) Added divideByType to set default for all sources
• (main) Added divideByUser to set default for all sources
• (main) Added divideByYear to set default for all sources
• (main) Added divideFoldersUseID to set default for all sources
• (main) Added filters to set default for all sources
• (main) Added historyTyping to set default for all sources
• (main) Added ignoreBots to set default for all sources
• (main) Added reactWhenDownloadedEmoji to set default for all sources
• (main) Added save to set default for all sources
• (main) Added saveAudioFiles to set default for all sources
• (main) Added saveImages to set default for all sources
• (main) Added saveOtherFiles to set default for all sources
• (main) Added savePossibleDuplicates to set default for all sources
• (main) Added saveTextFiles to set default for all sources
• (main) Added saveVideos to set default for all sources
• (main) Added scanEdits to set default for all sources
• (main) Added sendErrorMessages to set default for all sources
• (main) Added sendFileCaption to set default for all sources
• (main) Added sendFileDirectly to set default for all sources
• (main) Added sendFileToChannel to set default for all sources
• (main) Added sendFileToChannels to set default for all sources
• (main) Removed allowSkipping
• (main) Removed asyncHistory
• (main) Removed checkPermissions
• (main) Renamed allowGlobalCommands to allowGeneralCommands
• (main) Renamed autorunHistory to autoHistory
• (main) Renamed autorunHistoryBefore to autoHistoryBefore
• (main) Renamed autorunHistorySince to autoHistorySince
• (main) Renamed debugOutput to debug
• (main) Renamed filterDuplicateImages to duplo
• (main) Renamed filterDuplicateImagesThreshold to duploThreshold
• (main) Renamed numberFormatEuropean to europeanNumbers
• (main) Renamed presenceOverwrite to presenceLabel
• (main) Renamed presenceOverwriteDetails to presenceDetails
• (main) Renamed presenceOverwriteState to presenceState
• (main) Renamed inflateCount to inflateDownloadCount
• (sources) Added divideByMonth
• (sources) Added divideByYear
• (sources) Added duplo, duploThreshold
• (sources) Added sendAutoHistoryStatus
• (sources) Added sendFileCaption
• (sources) Added sendHistoryStatus
• (sources) Added user, users, category, categories, categoryBlacklist
• (sources) Removed overwriteAllowSkipping
• (sources) Renamed autorunHistory to autoHistory
• (sources) Renamed autorunHistoryBefore to autoHistoryBefore
• (sources) Renamed autorunHistorySince to autoHistorySince
• (sources) Renamed divideFoldersByChannel to divideByChannel
• (sources) Renamed divideFoldersByServer to divideByServer
• (sources) Renamed divideFoldersByType to divideByType
• (sources) Renamed divideFoldersByUser to divideByUser
• (sources) Renamed overwriteEmbedColor to embedColor
• (sources) Renamed overwriteFilenameDateFormat to filenameDateFormat
• (sources) Renamed overwriteFilenameFormat to filenameFormat
• (sources) Renamed updatePresence to presenceEnabled
• (filters) Added blockedFilenames, allowedFilenames
• (filters) Added blockedReactions, allowedReactions

WORKING ON A v2.0.0 REWRITE, JOIN DISCORD FOR BETA https://discord.gg/6Z6FJZVaDV

• Upgraded from Go 1.15 to 1.19 (Latest)
• Upgraded dependencies
• Modified discordgo to fix an issue selfbot users encountered when logging in
• Fixed emojis command crashing
• Added settings.reactWhenDownloadedHistory to toggle reactions for processing history messages, false by default.
• Renamed settings.channels[].errorMessages to sendErrorMessages
• Settings output with debugOutput now strips credentials

Currently working on a rewrite as a v2.0.0 with a modified version of the latest discordgo that restores selfbot functionality with no need to have multiple versions of the bot for genuine bots to take advantage of latest api features.

LATEST-API VERSION IS FOR GENUINE DISCORD BOTS ONLY, NOT SELFBOTS

• Improved connection recovery (reconnecting to the bot gateway after a disruption).
• Added config.exitOnBadConnection (bool, false by default). Exits bot upon connection disruption, good for Process Managers or Docker containers.
• Added config.discordTimeout (seconds, 180 by default). Overrides the bot clients timeout in seconds.
• Fixed history since & before tags using UTC for time instead of local time.

LATEST-API VERSION:

• Upgraded to discordgo v0.26.1 using discord's v9 api versus the v6 api the master branch uses. This means this version will ONLY work for actual Discord Bot Applications, not user accounts / selfbots.
• Rudimentary thread support, treats thread messages exactly the same as the channel they originate from.

• History process no longer deletes cache file upon completion, meaning re-running history for that channel should only search from the latest messages. If you wish to re-process all older messages, find the cache file under */cache/history/ and delete it. If desired this will become optional in settings for future releases.
• Added autorunHistoryBefore and autorunHistorySince (as well as overwriteAutorun... for server/channel specific overrides) to specify date ranges for history running upon launch. Follows YYYY-MM-DD format.
• Added {{fileType}}, {{nanoID}}, {{shortID}} keys to filename formatting. Nano ID is a 21 character unique string generated randomly upon download, Short ID is the same but 10 characters. -- Big thanks to @Twinki14 for this!
• Fixed an issue with Twitter media direct links not working properly.
• Fixed an issue with BlockedRoles/AllowedRoles not working properly.
• History command now solely listens to designated bot admins, formerly allowed discord users with admin permissions to run history.
• Implemented discordgo vulnerability fix from newer build
• Support for vxtwitter.com

Removes filterDuplicateImages debug log

• Fixed response messages being broken for selfbots/userbots
• Added channel/server settings sendFileToChannel (and sendFileToChannels for multiple channels) to forward/crosspost detected files into another channel.
• Added filenameFormat alongside the existing filenameDateFormat in general settings, as well as overwriteFilenameFormat for channels/servers. Formats the entire filename, default is "{{date}} {{file}}". Currently supports {{date}}, {{file}}, {{messageID}}, {{userID}}, {{username}}, {{channelID}}, {{serverID}}, {{message}}.
• Added divideFoldersUseID to channels/servers, uses ID for divideFoldersBy_ settings rather than name, good for sources that change names often.
• Added save setting to channels/servers (enabled by default, of course) to not actually download detected files
• Renamed blacklistChannels to serverBlacklist
• Added support for fxtwitter (twitter link alternative for embedding), and twitter mobile links
• Trims filenames that exceed character limits
• filterDuplicateImages logs score comparisons when debugOutput is enabled
• Fixed bot not responding to selfbot commands

• Added global reactWhenDownloaded, sets the default for the server/channel equivelant of reactWhenDownloaded
• Added reactWhenDownloadedHistory to server/channel configs, default is false, toggles reactions for downloaded messages processed in history
• Added channels alternative for adminChannels, to have multi-channel configs
• Ignores files if they return a 404 status code
• Improved Server/Channel Validation upon discord login
• History completely ends cataloging channel when it's exceeded the specified range
• Fixed history command before range not working
• Fixed sourceConfig.Enabled not working for history when false
• Tiny tiny fixes
• Bot will log all discord accessible servers+channels to cache/constants.json every login

• Fixed settings being unable to generate default file upon first launch
• Cleanup