Setting Up A PyTorch Pipeline To Track And Analyze Twitter Data
I spent about three months building a system that streams Twitter data, runs inference through fine-tuned PyTorch models, and stores the results for downstream analysis. The whole thing started because I needed to monitor sentiment around model releases in real time instead of waiting for weekly aggregation reports. What follows is exactly how I set it up, the problems I ran into, and the things I would do differently now. The core idea is straightforward. You pull tweets using the Twitter API v2, run them through a text classification or sentiment model built in PyTorch, and log the outputs. The "magic" people talk about is mostly just making sure the pieces fit together without memory blowing up on you during inference. I will get to the actual code structure in a moment. First, the hardware consideration. Running a BERT-sized model on CPU for a continuous stream of tweets is slow and impractical. I moved to a single A10G GPU instance on AWS and saw inference latency drop from roughly 85 milliseconds per tweet down to about 8 milliseconds. That matters when you are processing thousands of tweets per hour. If you do not have GPU access, consider using a distilled model like TinyBERT or ONNX Runtime with CUDA acceleration as a middle ground. It will not be as fast, but it keeps costs reasonable.
For the data pipeline, I used the Twitter API v2 with filtered stream endpoints. The key detail most people miss is that v2 rate limits are calculated per endpoint, not per app. I was hitting limits on the search endpoint while the stream endpoint sat half-empty because I had not separated my credentials correctly between the two operations. Use different bearer tokens or split your requests across separate apps if you need both search and streaming simultaneously. Here is the basic architecture I ended up running: 1. A Python script consuming the Twitter filtered stream using tweepy or the official Twitter SDK. It filters by keywords, languages, and engagement thresholds. I only kept tweets with over fifty retweets or replies to reduce noise. That cut my dataset size by about seventy percent without losing meaningful signal.
2. A preprocessing layer that strips URLs, normalizes emojis, removes excess whitespace, and truncates tweets to five hundred tokens. PyTorch tokenizers handle this cleanly. I use the DistilBERT tokenizer because it matches my model and runs faster than the full BERT tokenizer on CPU fallback. 3. The inference model itself, a fine-tuned DistilBERT classifier trained on a custom dataset of labeled tech-related tweets. I trained it using Hugging Face Transformers with a learning rate of twoe-5, batch size of thirty-two, and twelve epochs. Validation accuracy settled at about eighty-four percent. It is not perfect, but it catches the patterns that matter for my use case. 4. A storage layer writing results to PostgreSQL with JSONB columns for the model outputs. This made it trivial to query trends by date, sentiment distribution, and keyword frequency without re-running inference.
Get the Full Details
One specific problem I hit that took me two days to solve involved batch inference and CUDA memory fragmentation. When I processed tweets in batches of sixty-four, the GPU memory usage grew unpredictably until the process crashed with a CUDA out-of-memory error. The issue was not the batch size itself but how PyTorch caches memory allocations during repeated forward passes. I fixed it by adding torch.cuda.empty_cache() after each inference batch and switching to checkpointing on the model. That reduced peak memory usage by roughly forty percent and stabilized the process for days at a time without manual intervention. Another detail worth mentioning is that the Twitter API free tier gives you only fifteen thousand tweets per month. I underestimated this completely in the first week and burned through the quota reading only niche tech keywords. Switching to the academic research endpoint gave me full archive access for my project, but it added a twenty-four hour delay to tweet visibility. If real-time matters more than historical depth, the paid Basic tier at one hundred thousand tweets per month is the minimum viable option. The Enterprise tier is overkill unless you need ten million tweets or more. For the actual PyTorch implementation, here is the inference function I settled on after iterating through several versions:
def predict_tweets(model, tokenizer, texts, device): model.eval() tokens = tokenizer(texts, padding=True, truncation=True, max_length=512, return_tensors='pt').to(device)
with torch.no_grad(): outputs = model(tokens) predictions = torch.softmax(outputs.logits, dim=1).cpu().numpy()
return predictions This looks simple and it is. The important parts are torch.no_grad() to prevent gradient tracking during inference, moving the tokenized inputs to the correct device before the forward pass, and softmax output for interpretable probabilities. Without the no_grad context manager, I saw memory usage double during long inference runs because PyTorch was storing computation graphs for tensors that never needed backpropagation. If you want the model weights and tokenizer from my project, I uploaded them to Hugging Face under a permissive license. You can find the repository by searching for the label "tech-sentiment-distilbert-finetuned" on the Hugging Face hub. The training script, configuration files, and evaluation metrics are all included. I also pushed a Dockerfile that packages the inference service with its dependencies.
There are real limitations to this setup that I want to be clear about. The sentiment model performs poorly on sarcasm and irony, which are common in tech Twitter discourse. About twelve percent of my manually sampled outputs were clearly misclassified for this reason. You can improve this with adversarial training or by adding a sarcasm-specific fine-tuning dataset, but both approaches require labeled data you may not have. Another bottleneck is the Twitter API itself, which changed its access policies multiple times over the past year. Features I relied on in 2024 became restricted or required different authentication flows in 2025. Budget for maintenance work regardless of how clean your code is. A practical alternative if you do not need raw Twitter data is to use third-party aggregators like Brandwatch or Sprout Social, which provide pre-computed sentiment scores and trend dashboards. They cost money and offer less customization, but they eliminate the entire infrastructure maintenance burden. If you are building this for a startup prototype, skip the custom pipeline entirely and use their APIs until you hit a scale or specificity wall. Most teams do not reach that wall. The code structure for the full pipeline is available on GitHub. The README includes environment setup instructions, a sample configuration file for the Twitter API credentials, and a notebook showing how to reproduce the inference results. I tested everything on Python 3.11 with PyTorch 2.2.1, transformers 4.40.0, and tweepy 4.14.0. Versions older than these will work but may require dependency patching.
If you decide to build this yourself, start small. Get a stream working first without any model attached. Log tweets to a file and verify that your API credentials, filters, and rate limit handling are correct. Then add the preprocessing layer and test it on a static dataset before connecting it to the live stream. Adding the model last means you can debug data flow issues without GPU complications interfering. I wish I had followed this order myself. The two days I lost to the CUDA memory issue would have been avoided if the pipeline had been stable before I introduced the inference component. The whole system runs on a modest g5.xlarge EC2 instance at approximately eighty dollars per month including GPU time and PostgreSQL. Your costs will vary depending on tweet volume and inference frequency. I tracked mine using AWS Cost Explorer and adjusted the batch size and polling intervals to stay within budget while maintaining acceptable latency. The final processing delay from tweet publication to logged prediction was about two seconds on average, which was sufficient for the use case I built it for.
