SQL Streaming Module – Documentation & Evolutions

Context

Bouygues Telecom's internal ETL includes a SQL streaming module built on top of Apache Flink. It exposes a SQL-derived language letting data engineers define streaming jobs (source → sink) without writing Flink code directly. The module was functional but not production-ready: connectors lacked documentation and the available options were invisible to users.

Approach

• ·Deep-dived into the internal ETL architecture and the streaming module internals to understand the full connector system
• ·Built a Maven Java plugin that parses the module's source code, extracts annotations, and auto-generates Markdown documentation
• ·Deployed a Docusaurus static site to expose the generated docs in a navigable, user-friendly format
• ·Integrated the documentation pipeline into the existing CI/CD chain
• ·Worked on streaming evolutions: error handling improvements and robustness to malformed data

Solution

An automated documentation system that keeps the streaming module's connector reference always in sync with the source code, deployed via CI/CD — giving data engineers a reliable, up-to-date reference to write streaming jobs autonomously.