PyFlink - datagen connector

 

from pyflink.table import TableEnvironment, EnvironmentSettingssettings = EnvironmentSettings.in_streaming_mode()
t_env = TableEnvironment.create(settings)t_env.execute_sql("""CREATE TABLE MySource (id INT,name STRING) WITH ('connector' = 'datagen')
""")t_env.execute_sql("SELECT * FROM MySource").print()

For the above code, where will the table MySource be stored?

 

Short answer:
👉 MySource is not stored anywhere.
It is a logical (virtual) table, backed by a data generator, and exists only in the running Flink job.

---

Why MySource is not stored

1. datagen is a source, not storage

'connector' = 'datagen'

• datagen generates data on the fly

• No files, no database, no persistent storage

• Rows are created in memory at runtime

---

2. Table API tables are metadata + connectors

In Flink, a table consists of:

• Schema (columns, types)

• Connector definition

• Optional catalog metadata

It does not imply physical storage unless the connector provides it.

---

What actually happens at runtime

SELECT * FROM MySource

• Flink starts the job

• datagen produces rows (random or sequential)

• Rows flow through the pipeline

• .print() writes results to stdout

Once the job stops:

• MySource disappears

• Generated data is lost

---

Where the table definition lives

Scope	Exists where
Table schema	Flink job (JVM memory)
Generated rows	TaskManager memory
Persistence	❌ None

Since you didn’t configure:

• A catalog (Hive, JDBC, filesystem)

• A sink (Kafka, filesystem, DB)

Nothing is saved.

---

How tables can be stored persistently

1. Use a catalog (for metadata)

t_env.execute_sql("""
CREATE CATALOG my_catalog WITH ('type' = 'hive',...
)
""")

2. Use a persistent connector

Examples:

• Filesystem (HDFS / S3)

• Kafka

• JDBC

• Iceberg / Hudi / Delta

Example sink:

CREATE TABLE MySink (id INT,name STRING
) WITH ('connector' = 'filesystem','path' = '/tmp/output','format' = 'csv'
)

---

Key takeaway

In Flink, tables are logical abstractions.
They are only “stored” if the connector stores data.

Your MySource table:

• ❌ Not stored on disk

• ❌ Not stored in a database

• ✅ Generated dynamically

• ✅ Exists only during job execution

 

(airflow-venv) frank@ZZHPC:~/zproject$ flink run -py my_job.py
| +I |   549674300 | ac2be66685e79f7152901d0d61a... |
| +I | -1259077681 | 7773378ffd59e62096726aa8b26... |
| +I |  -112218307 | 9f1e979db8abf8cc2e4c1d8310d... |
| +I | -1392104357 | a7dba2cd32fe9c48a759d8a886d... |
| +I | -1516081769 | aa0ca72fb671b46c2b57545cf71... |
| +I |   380579405 | 861ce6e96d244dd6cebd243ec37... |
| +I |  1748819335 | bd162fcbd6c39ad5e47f35055b7... |
| +I | -1648463948 | 37e9643a96a4de8a355928159ea... |
| +I | -1925178175 | 20de30b7135ac186f2461130b11... |
| +I | -1913158356 | c8709a8bbacc2373c24927df314... |
| +I |  -844590625 | c6aae7818d2ddf6f9fd6cdcb836... |
| +I |   551639609 | 2279a5172fcb034030d3182d1fd... |
| +I |   115702922 | 407f6ad6dd3b41b8f65315a9113... |
| +I | -1249309820 | 04fa0efbe2720b23ac369610af2... |
| +I |   307810094 | 4f8211ad2a2daff8142a15ae497... |
| +I |   642408656 | 63e9f697a28324a551124c41f7d... |
| +I |   -26470531 | 0572620b06e7b0e96e722849e85... |
| +I |  -237334693 | 588eb2a24c70a0f0b19f592f904... |
| +I |   680441998 | c2ad60758acc6ea210e54112e02... |
| +I |  1629320224 | d08a9e4e89a873256b0c11f1291... |
......