Skip to content

Swadhin Biswas

Data Engineer · Databricks, AWS, Spark, Kafka, dbt, Python

Dhaka, Bangladesh · Remote · Open to relocation

swadhinbiswas.dev@gmail.com+880 1759-877314linkedin.com/in/swadhinbiswasgithub.com/swadhinbiswasswadhin.cv

Summary

Data Engineer with 5 years in software, 3+ of them on production data and backend systems. I design production-grade data pipelines, event-driven ingestion systems, and data/ML infrastructure using Python, SQL, Kafka, dbt, DuckDB, Databricks (Spark, Delta Lake, Unity Catalog, MLflow, Workflows), and AWS (Glue, Redshift, Athena). Co-founded The Boring Rat, scaled it to 1M+ users on donation-funded growth, and led it to an acquisition in 2025. Currently building OpenCodeHub, a self-hosted developer platform, alongside three production lakehouses: EU Air Traffic, a Kafka-to-DuckDB pipeline that refreshes over 2,000 live planes across Europe every 15 minutes; TerraSentinel, a native Databricks satellite anomaly platform with a Unity Catalog warehouse and MLflow registry; and EuroStream, a GDPR-compliant commerce lakehouse with sub-second Article 17 erasure.

Experience

Founder · OpenCodeHub

Open-source, self-hosted alternative to GitHub for source control, CI/CD, code review, and package registries. Solo build, self-hosted and in active daily use; public launch pending.

  • Self-hosting the platform and using it to manage my own repositories and projects, serving as the primary dogfooding environment ahead of public launch.
  • Implementing core backend services for repository management, Git transport protocols, authentication/authorization, and event-driven workflow automation.
  • Designing the platform's data layer, including audit trails, usage telemetry, and repository insights, ahead of public launch.
  • Owning the full stack solo: architecture, backend, databases, caching, storage, and containerized deployment.

Co-Founder & Lead Data/Backend Engineer · The Boring Rat (Acquired)

Developer platform and community operated on donation-based funding, with no paid marketing or venture capital. Reference available from my co-founder.

  • Co-founded and scaled the platform to 1M+ active users.
  • Owned the data and backend stack end to end, building real-time event ingestion pipelines, the analytics warehouse, and production infrastructure.
  • Designed and operated MLOps workflows powering the platform's data and ML features, from pipeline development through deployment.
  • Led the company through its acquisition in November 2025 under NDA.

Backend & Automation Engineer · Freelance

  • Delivered 8+ custom software solutions for international clients, primarily SaaS and e-commerce teams in the US and UK, spanning backend systems, data platforms, automation tools, and business applications.
  • Designed and maintained PostgreSQL databases, data models, and REST APIs supporting transactional and analytical workloads, including Django APIs that served 10K+ requests a day.
  • Built large-scale web scraping and data ingestion systems that collected, processed, and stored data from public websites, APIs, and third-party services.
  • Implemented data quality checks, monitoring, scheduling, and automation processes to improve reliability and reduce manual intervention.
  • Worked directly with clients to translate business requirements into scalable software and data solutions, cutting page load times by 40% through React optimization.

Selected Projects

EU Air Traffic

Python, Apache Kafka, dbt, DuckDB, FastAPI, MotherDuck, Turso, AWS (Lambda, S3, Glue, Athena)

A solo-built, unattended production lakehouse for live and historical European airspace.

  • Architected a medallion (Bronze/Silver/warehouse) lakehouse behind a 5-topic Kafka event bus, streaming ADS-B positions, OpenSky flight movements, AirLabs schedules, and METAR/TAF weather in real time; refreshes ~2,000 live-tracked aircraft on a 15-minute cadence.
  • Built an incremental Silver layer and star-schema DuckDB warehouse with 103 dbt models/tests, deduplicating cross-provider records (OpenSky vs. AirLabs) on callsign/date/endpoint to eliminate double-counted flights.
  • Designed a dual-serving architecture (full warehouse on MotherDuck, a scoped read-only copy on Turso with automatic failover when a database hits its quota) and enforced per-provider rate and credit budgets in code, keeping a 2-core VPS collector running unattended. Backed by 120+ pytest tests and an MIT-licensed dataset on Hugging Face with a Zenodo DOI.
  • Extended the pipeline onto AWS: a Lambda job exports the Bronze-layer DuckDB history as Parquet into an S3 raw zone, a Glue Crawler catalogs it, and a PySpark Glue ETL job cleans the data and writes curated fact/dimension/aggregate Parquet into a separate S3 curated zone queryable directly through Amazon Athena.
  • Provisioned the AWS stack with Terraform, wired GitHub Actions to run dbt tests against it, and set AWS Budget cost alerts with teardown after each demo run, keeping the serverless extension cheap to spin up and validate independently of the always-on collector.

TerraSentinel

Python, Databricks, Apache Spark, Delta Lake, Unity Catalog, dbt, DuckDB, MLflow, Hugging Face, Turso

A $0-cost anomaly-detection platform over free satellite and sensor data, shipped as a hybrid free-tier/Databricks lakehouse rather than a notebook.

  • Built a free-tier Bronze/Silver/Gold lake on DuckDB (117 dbt nodes, 7 gold marts) from four public sources (NASA FIRMS fire detections, Sentinel imagery, NOAA/NSIDC sea-ice and SST, ENTSO-E grid load), scheduled on GitHub Actions and served via Turso for edge reads.
  • Mirrored it as a Databricks Asset Bundle: six Workflows jobs on Quartz crons CI-tested against the GitHub Actions cadence, declarative Spark/Delta Lake pipelines, and Unity Catalog DDL. It's deployed paused by design, so handover between schedulers is a one-line, side-effect-free flip.
  • Trained an unsupervised IsolationForest on 27 strictly causal features with MLflow tracking and a Hugging Face model registry, batch-scoring predictions into the serving tables; validated with a synthetic-anomaly CI fixture and 592 tests, and documented the trade-offs as ADRs.

EuroStream

Python, Databricks Asset Bundles, Delta Live Tables, Apache Spark, Delta Lake, Unity Catalog, Kafka, DuckDB, FastAPI, Turso

A GDPR-native, sovereign medallion lakehouse for European commerce, built to make EU data-protection obligations architectural invariants rather than policy.

  • Built a Kafka-fed medallion (Bronze/Silver/Gold) lakehouse that enforces GDPR at the ingestion boundary: deterministic salted-SHA-256 pseudonymization at the Silver layer and Unity Catalog grants/tags that keep clear-text PII (IBAN, email, IP) from ever escaping to external query layers.
  • Implemented a six-layer Article 17 erasure cascade performing sub-second physical and cryptographic deletion across every storage tier, with a suppression registry that intercepts replayed events and a benchmark job verifying zero ghost records in materialized aggregates.
  • Deployed on Databricks as an Asset Bundle (Delta Live Tables Bronze/Silver/Gold pipelines in a dependency-ordered Workflows DAG, with idempotent Delta MERGE ingest) and shipped a DuckDB analytical surface, Turso serving copy, FastAPI read API, and CLI. Covered by 59 tests, strict mypy, and ruff in CI, with a JOSS-format design paper.

Skills

  • Languages: Python, SQL
  • Frameworks: FastAPI
  • Data Engineering: Apache Kafka, Apache Spark, Databricks (Asset Bundles, Workflows, Delta Live Tables, Unity Catalog), Delta Lake, dbt
  • MLOps: MLflow (tracking & model registry), scikit-learn
  • Databases: PostgreSQL, DuckDB, Turso (libSQL)
  • Cloud & Infrastructure: AWS (Glue, Athena), Cloudflare Pages, Docker, Git, GitHub Actions (CI/CD), Terraform
  • Languages Spoken: English (Fluent) · Bengali (Native) · Hindi (Working) · German (Learning)

Education

B.Sc. in Computer Science & Engineering · Daffodil International University

Dhaka, Bangladesh

Relevant Coursework: Data Structures & Algorithms, Database Management Systems, Operating Systems, Distributed Systems, Computer Networks, Software Engineering