Source plugin : File [Spark]
read data from local or hdfs file.
name | type | required | default value |
---|---|---|---|
format | string | no | json |
path | string | yes | - |
common-options | string | yes | - |
Format for reading files, currently supports text, parquet, json, orc, csv.
- If read data from hdfs , the file path should start with
hdfs://
- If read data from local , the file path should start with
file://
Source plugin common parameters, please refer to Source Plugin for details
file {
path = "hdfs:///var/logs"
result_table_name = "access_log"
}
file {
path = "file:///var/logs"
result_table_name = "access_log"
}