I recently downloaded my full history of tweets from Twitter (via the “Request your archive” button on the Settings page), which gives the data in CSV and JSON formats. The download comes with a simple web UI so you can browse locally, all your historical tweets, as well as perform basic searches. When I tried performing a wildcard search, it resulted in an unresponsive script warning and nobbled the browser tab.
Cue ElasticSearch
I’ve kept meaning to have another hack around with ElasticSearch for a while now, as I started looking into it in 2011 and was very impressed with it. So I decided to knock up a little .NET app to take the .js files provided by Twitter in the export, extract the tweet data and push into ElasticSearch. It seems like a good little starter project to learn more about ElasticSearch and actually could give me some useful insights into all the tweets I’ve made. I’m sure I’m going to find some really good links I tweeted and then couldn’t find again!
At the moment there’s a very basic GUI (and I mean basic!) exposing the functionality to import the tweet data into ElasticSearch. I plan to stick a basic search GUI together too and then…well, I’m not quite sure yet, but I’m sure as time goes on I’ll see more ways to extract interesting information from my tweet history.
The ElasticTweets project as I’ve called it, can be found on GitHub here - feel free to use and abuse it, and let me know how you get on!