Uses Twitter archive to visualize your Twitter network based on your replies, quote tweets and direct messaging history. Get DM stats with leaderboard based on message count, bar graph of your messaging history with your friends
MIT License
If you have already cloned, please do a git pull
Download your Twitter archive.
More (3 dot button) > Settings and Privacy > Your Account > Download an archive of your data
.Clone the repository:
git clone https://github.com/sankalp1999/twitter-circle.git
Copy your Twitter archive (extracted folder, not the zip file) into the project folder and rename the extracted folder to twitter-archive
. You have to ensure the archive is on the root folder of the project.
Install Node.js (for Linux and MacOS)
Install project dependencies
npm install
This command will install all the necessary packages listed in the package.json
file.
Set up the project
./setup.sh
For Windows, you will require git bash or WSL to run above bash script.
Start the application
npm start
This command will start the Twitter Circle application. Share screenshots of your top 100 now! If you liked it, please star the repo.
First step if not working, do a git pull
and just re-run the script ./setup.sh
If still not working, raise an issue or contact me https://twitter.com/dejavucoder
Update 16th June
Most problems will happen at the scraping stage. I don't directly scrape profile pics from Twitter. I currently scrape from muskviewer.com. You should be able to scrape once properly. If you try a second scrape immediately, then you may get rate limited.
Earlier I was using twstalker.com which now has cloudfare bot detection. If you could help me get around this, I am open to a PR.
The frontend tweet webviewer website may be down. you can check by twstalker.com/your_user_name. In this case, you can try
later or try changing line 146 in pfp_fetch_and_id_correction.js
. if (isReachablePrimary)
to if (false)
Profile pictures not rendering
a. Browser dependencies are missing - check https://pptr.dev/troubleshooting
For Linux, WSL etc. check here.
Error may look like below (it's from WSL)
UnhandledPromiseRejectionWarning: Error: Failed to launch the browser process!
/your_username/.cache/puppeteer/chrome-linux-122.0.6261.69/chrome-linux/chrome: error while loading shared libraries: libatk-1.0.so.0: cannot open shared object file: No such file or directory
TROUBLESHOOTING: https://pptr.dev/troubleshooting
b. Browser launch process fail because chromium path not set (Linux, M1 Macs)
Find your chromium path and please set it like below example
around line 211 in pfp_fetch_and_id_correction.js
const browser = await puppeteer.launch({ headless: true, args: ['--no-sandbox', '--disable-setuid-sandbox'], executablePath: 'usr/bin/chromium-browser' })
Chrome or Safari are recommended. Edge has CORS issue.
Don't forget to share screenshots of your top 50, 100 whatever you feel like
Tag your inner circle in reply
Zoom out browser enough to let the images fit in and please exclude slider for better screenshot.
If you are feeling courageous, I dare you to share your DM stats leaderboard screenshots.
Tried to keep complexity and dependencies at minimum
account.js
- Details of your account like accountId and userhandle/screennametweets.js
- All your tweets (normal tweet, replies, quote tweets) with data like reply mentions, quote tweet url, text, media urldirect-messages.js
- All your personal messages, no group chat messagesgraph TD
A[Extract mentions and quote tweets] --> B[Create user ID to screen name mapping]
A --> C[Calculate weighted scores based on frequency and recency]
D[Read direct messaging data] --> E[Extract DM data and calculate basic stats]
D --> F[Compute DM weights]
F --> G[Combine DM weights with mention weights]
G --> H[Take top N users]
H --> I[Fetch profile picture CDN URLs for top N users]
H --> J[Attempt to correct missing user IDs]
J --> K[Update ID-to-username mapping]
K --> L[Create interactive visualization of Twitter circle]
K --> M[Display ranking of friends based on combined weights]
D --> N[Process data to count messages per month for past 5 years]
N --> O[Prepare data for visualizing DM history graph]
O --> P[Display table of DM statistics per recipient]
P --> Q[Display bar graph of monthly message counts for selected recipient]
style A fill:#ff9980
style B fill:#ff9980
style C fill:#ff9980
style D fill:#80b3ff
style E fill:#80b3ff
style F fill:#80b3ff
style G fill:#b3d9ff
style H fill:#cccccc
style I fill:#cccccc
style J fill:#cccccc
style K fill:#e6e6e6
style L fill:#b3ffb3
style M fill:#b3ffb3
style N fill:#ffff99
style O fill:#ffff99
style P fill:#ffcc99
style Q fill:#ffcc99
graph TD
A(Read tweets.js) --> B(Extract mentions and quote tweets)
B --> C(Create user ID to screen name mapping)
B --> D(Calculate mentions weights)
D --> E(Save weights to mentionsCountWeighted.json)
F(Read direct-messages.js) --> G(Extract DM data)
G --> H(Calculate DM stats)
H --> I(Calculate DM weights)
I --> J(Combine DM weights with mentions weights)
J --> K(Save combined weights to sortedCombinedWeights.json)
K --> L(Read sortedCombinedWeights.json)
L --> M(Fetch top N users' profile pictures)
M --> N(Correct missing user IDs)
N --> O(Save updated data to final_weights_with_pics.json)
G --> P(Process DM data for visualization)
P --> Q(Save DM stats with chart data to dm_final_stats_with_chart.json)
O --> R(Load data in index.html for D3.js visualization)
O --> S(Load data in leaderboard.html for ranking display)
Q --> T(Load data in dm_stats.html for DM stats table)
T --> U(Click on a row to view DM history)
U --> V(Load specific recipient's data in chart_draw.html for DM history graph)
classDef script fill:#f9f,stroke:#333,stroke-width:4px;
classDef data fill:#ccf,stroke:#333,stroke-width:2px;
classDef webpage fill:#cfc,stroke:#333,stroke-width:2px;
class A,B,F,G script;
class C,D,E,H,I,J,K,N,O,P,Q data;
class R,S,T,U,V webpage;
extract_mentions_and_dump.js
:
tweets.js
from the user's Twitter archive.user_mentions_screen_name_mapping.json
.mentions_count_folder/mentionsCountWeighted.json
.Scores are based on your replies and quote tweets. I sum up the mentions and apply a weighing mechanism based on time difference to ensure the relevance of interactions. Recent interactions get slightly more weightage. This is because we perceive people we interacted with recently to be closer to us - the time weight heuristic is provided to account for the recency bias
The mapping between user IDs and screen names is to avoid scraping. This mapping is required as the direct messaging data contains only accountIds (and no usernames)
The mapping will work if you have replied to the person at least once otherwise their accountId won't be known. There is a scraping workaround but I avoid it as it takes time plus want to keep scraping at minimum. see utils/fetch_user_id_to_user_name.js
preprocess_direct-messages.js
:
direct-messages.js
from the user's Twitter archive.mentions_count_folder/mentionsCountWeighted.json
and saves the result to sortedCombinedWeights.json
.pfp_fetch_and_id_correction.js
:
sortedCombinedWeights.json
and takes the top N users.final_weights_with_pics.json
.dm_final_stats_processing.js
:
twitter-archive/data/direct-messages.js
.dm_final_stats_with_chart.json
.index.html
:
final_weights_with_pics.json
.leaderboard.html
:
final_weights_with_pics.json
.dm_stats.html
:
dm_final_stats_with_chart.json
.chart_draw.html
with the recipient's ID as a URL parameter.chart_draw.html
:
dm_final_stats_with_chart.json
.Just look at the data like Lain. Stare at it.
We get free mapping between name and id. It took me sometime to realise this. I had already written the scraper.
If you have replied to someone at least once, then you have a valid mapping.
@notfound_userid
in the DM stats table. These are accounts where we couldn't find a mapping between username and id from the reply mentions data. Since there combined weights do not end up in the topN, their pfp and banner is not fetched so they end up as not found.I can add some code to fetch banners (so i can get the userid) for top 200 dm stats also. It will require less than 200 ofc because most people you talk to in DM you have replied to them at least once on the timeline.
If you really want to find them, you can try going to https://twitter.com/intent/user?user_id=user_id or try https://twitter.com/i/user/user_id. You need to be logged in for this.
This project is licensed under the MIT License. See the LICENSE file for details.