Real Time Crypto Monitoring System
Solo Developer
A real time data pipeline that fetches, processes, and stores cryptocurrency exchange rates around the clock using Apache Kafka and Cassandra, deployed on AWS.
Overview
About the project
Banks, fintech firms, and traders need an up to date view of cryptocurrency markets for decisions like fraud detection, trading signals, and compliance reporting. Traditional batch jobs that run once a day introduce a delay that is unacceptable here, because by the time the data lands the moment has already passed.
I designed a three stage streaming architecture. An API consumer pulls live exchange rates for BTC, ETH, and XRP against USD and EUR from the Alpha Vantage API every 30 minutes, scheduled with APScheduler. A Kafka producer formats and publishes each reading as a message, decoupling data collection from storage so neither side blocks the other. A Kafka consumer then writes the structured records into Apache Cassandra, a database built for high throughput writes. The whole system runs on an AWS EC2 instance with continuous 24/7 uptime.
The project delivers a production pattern streaming pipeline that runs unattended. It demonstrates the exact ingestion to storage flow that financial institutions adapt for real time monitoring, anomaly detection, and reporting.
System Overview
At a glance
A real time streaming pipeline that pulls live cryptocurrency exchange rates, ships them through Apache Kafka, and lands them in Apache Cassandra for downstream use. The whole thing runs unattended on an AWS EC2 instance with 24/7 uptime and a clear separation between ingestion, transport, and storage.
How It Works
System Architecture
Alpha Vantage API
Real time cryptocurrency data source
Apache Kafka
Message streaming platform
Apache Cassandra
Distributed NoSQL database
Process Flow
How the project moves from start to finish, step by step.
Ingest
An API consumer fetches live exchange rates for BTC, ETH, and XRP against USD and EUR every 30 minutes.
Stream
A Kafka producer streams formatted cryptocurrency data to a topic for real time processing.
Store
A Kafka consumer processes the messages and inserts structured data into Cassandra for persistence.
System Breakdown
Streaming Architecture
Three independent stages, each free to fail or restart without affecting the others.
Producer Stage
API Consumer
Fetches live exchange rates from the Alpha Vantage API.
APScheduler Job
Runs the fetch every 30 minutes without manual intervention.
Message Serialiser
JSON encodes each reading before publishing to Kafka.
Streaming Stage
Kafka Topic
Holds in flight messages and decouples producer from consumer.
Bootstrap Server
Local Kafka broker handling produce and consume traffic.
Storage Stage
Kafka Consumer
Reads messages from the topic and prepares them for insert.
Cassandra Table
Persists structured rows with bid, ask, exchange rate, and timestamp.
Deployment Plane
AWS EC2 (t2.small)
Ubuntu host running every component continuously.
nohup Processes
Keeps producer and consumer running in the background across sessions.
Workflows
End to End Flow
Each interval the same three things happen, on schedule, without anyone watching.
Scheduled Ingestion
Purpose: Pull fresh exchange rates from the upstream API on a fixed cadence.
How it works
- APScheduler triggers the fetch every 30 minutes.
- The API consumer calls Alpha Vantage for BTC, ETH, and XRP against USD and EUR.
- The response is shaped into a JSON message with bid, ask, rate, and timestamp.
Message Streaming
Purpose: Hand readings to the storage layer without coupling the two.
How it works
- The Kafka producer publishes each message to the crypto_prices topic.
- The topic decouples the ingestion schedule from storage throughput.
- Backpressure is absorbed by Kafka rather than the API consumer.
Consume and Store
Purpose: Land every reading into a queryable store.
How it works
- The Kafka consumer subscribes to the topic and reads messages as they arrive.
- Each message is deserialised back into a structured row.
- The row is inserted into Cassandra with full timestamp and currency metadata.
Under The Hood
Technical Implementation
# Scheduled data fetching
scheduler = BlockingScheduler()
scheduler.add_job(
fetch_and_send_data,
"interval",
minutes=30,
)
# Kafka producer
producer = KafkaProducer(
bootstrap_servers="localhost:9092",
value_serializer=lambda v:
json.dumps(v).encode("utf-8"),
)# Kafka consumer
consumer = KafkaConsumer(
"crypto_prices",
auto_offset_reset="earliest",
value_deserializer=lambda x:
json.loads(x.decode("utf-8")),
)
# Cassandra insert
INSERT INTO crypto_prices VALUES
(%s, %s, %s, %s, %s, %s)Multi Currency Support
Tracks BTC, ETH, and XRP against USD and EUR at the same time.
Real Time Streaming
Automated 30 minute intervals driven by APScheduler.
Message Processing
Continuous listening to and processing of Kafka messages.
Persistent Storage
Structured storage with bid and ask prices and timestamps.
Data Model
Data Schema & Structure
What It Does
Features & Capabilities
Real Time Streaming
30 minute intervals with a continuous data flow.
AWS Deployment
A production ready EC2 instance with 24/7 uptime.
Secure Architecture
Environment variables and careful API key management.
Scalable Design
A modular architecture ready for horizontal scaling.
Running It
Deployment & Technology Stack
AWS EC2 Instance
t2.small
Ubuntu server running with 24/7 availability
Process Management
nohup
Background processes for continuous operation