taskmanager.data.bind-port |
(none) |
String |
The task manager's bind port used for data exchange operations. Also accepts a list of ports (“50100,50101”), ranges (“50100-50200”) or a combination of both. If not configured, 'taskmanager.data.port' will be used. |
taskmanager.data.port |
0 |
Integer |
The task manager’s external port used for data exchange operations. |
taskmanager.data.ssl.enabled |
true |
Boolean |
Enable SSL support for the taskmanager data transport. This is applicable only when the global flag for internal SSL (security.ssl.internal.enabled) is set to true |
taskmanager.network.compression.codec |
LZ4 |
Enum |
The codec to be used when compressing shuffle data. If it is "NONE", compression is disable. If it is not "NONE", only "LZ4", "LZO" and "ZSTD" are supported now. Through tpc-ds test of these three algorithms, the results show that "LZ4" algorithm has the highest compression and decompression speed, but the compression ratio is the lowest. "ZSTD" has the highest compression ratio, but the compression and decompression speed is the slowest, and LZO is between the two. Also note that this option is experimental and might be changed in the future.
Possible values: |
taskmanager.network.detailed-metrics |
false |
Boolean |
Boolean flag to enable/disable more detailed metrics about inbound/outbound network queue lengths. |
taskmanager.network.hybrid-shuffle.external-remote-tier-factory.class |
(none) |
String |
The option configures the class that is responsible for creating an external remote tier factory for hybrid shuffle. If configured, the hybrid shuffle will only initialize the specified remote tier according to the given class name. Currently, since the tier interfaces are not yet public and are still actively evolving, it is recommended that users do not independently implement the external remote tier until the tier interfaces are stabilized. |
taskmanager.network.hybrid-shuffle.remote.path |
(none) |
String |
The option is used to configure the base path of remote storage for hybrid shuffle. The shuffle data will be stored in remote storage when the disk space is not enough. Note: If the option is configured and taskmanager.network.hybrid-shuffle.enable-new-mode is false, this option will be ignored. If the option is not configured and taskmanager.network.hybrid-shuffle.enable-new-mode is true, the remote storage will be disabled. |
taskmanager.network.memory.read-buffer.required-per-gate.max |
(none) |
Integer |
The maximum number of network read buffers that are required by an input gate. (An input gate is responsible for reading data from all subtasks of an upstream task.) The number of buffers needed by an input gate is dynamically calculated in runtime, depending on various factors (e.g., the parallelism of the upstream task). Among the calculated number of needed buffers, the part below this configured value is required, while the excess part, if any, is optional. A task will fail if the required buffers cannot be obtained in runtime. A task will not fail due to not obtaining optional buffers, but may suffer a performance reduction. If not explicitly configured, the default value is Integer.MAX_VALUE for streaming workloads, and 1000 for batch workloads. If explicitly configured, the configured value should be at least 1. |
taskmanager.network.netty.client.connectTimeoutSec |
120 |
Integer |
The Netty client connection timeout. |
taskmanager.network.netty.client.tcp.keepCount |
(none) |
Integer |
The maximum number of keepalive probes TCP should send before Netty client dropping the connection. Note: This will not take effect when using netty transport type of nio with an older version of JDK 8, refer to https://bugs.openjdk.org/browse/JDK-8194298. |
taskmanager.network.netty.client.tcp.keepIdleSec |
(none) |
Integer |
The time (in seconds) the connection needs to remain idle before TCP starts sending keepalive probes. Note: This will not take effect when using netty transport type of nio with an older version of JDK 8, refer to https://bugs.openjdk.org/browse/JDK-8194298. |
taskmanager.network.netty.client.tcp.keepIntervalSec |
(none) |
Integer |
The time (in seconds) between individual keepalive probes. Note: This will not take effect when using netty transport type of nio with an older version of JDK 8, refer to https://bugs.openjdk.org/browse/JDK-8194298. |
taskmanager.network.partition-request-timeout |
10 s |
Duration |
Timeout for an individual partition request of remote input channels. The partition request will finally fail if the total wait time exceeds twice the value of taskmanager.network.request-backoff.max . |
taskmanager.network.request-backoff.initial |
100 |
Integer |
Minimum backoff in milliseconds for partition requests of local input channels. |
taskmanager.network.request-backoff.max |
10000 |
Integer |
Maximum backoff in milliseconds for partition requests of local input channels. |
taskmanager.network.retries |
0 |
Integer |
The number of retry attempts for network communication. Currently it's only used for establishing input/output channel connections |
taskmanager.network.sort-shuffle.min-buffers |
512 |
Integer |
Minimum number of network buffers required per blocking result partition for sort-shuffle. For production usage, it is suggested to increase this config value to at least 2048 (64M memory if the default 32K memory segment size is used) to improve the data compression ratio and reduce the small network packets. Usually, several hundreds of megabytes memory is enough for large scale batch jobs. Note: you may also need to increase the size of total network memory to avoid the 'insufficient number of network buffers' error if you are increasing this config value. |
taskmanager.network.tcp-connection.enable-reuse-across-jobs |
true |
Boolean |
Whether to reuse tcp connections across multi jobs. If set to true, tcp connections will not be released after job finishes. The subsequent jobs will be free from the overhead of the connection re-establish. However, this may lead to an increase in the total number of connections on your machine. When it reaches the upper limit, you can set it to false to release idle connections. Note that to avoid connection leak, you must set taskmanager.network.max-num-tcp-connections to a smaller value before you enable tcp connection reuse. |