Amazon S3
Connection Settings
Access Key
IAM created user access key with S3 permission.
Secret Key
IAM provided secret key.
Region
Region of the Amazon Kinesis Data Streams instance (e.g., us-east-1)
Input Settings
Type
Specifies the type of input operation. List will return the a set of directories and files at a specified path. Read will retrieve the data associated with a specific file.
List
Bucket Name
The name of the bucket to read data from.
Key
The name of the file to read.
Recursive
Specifies the set of files to list. When false, only immediate children of the directory are returned. When true, all files and directories under this path are included recursively.
Read
Bucket Name
The name of the bucket to read data from.
Key
The name of the file to read.
Note: Retrieving files stored in the S3 Glacier Flexible Retrieval storage class, the S3 Glacier Deep Archive storage class, the S3 Intelligent-Tiering Archive Access tier, or the S3 Intelligent-Tiering Deep Archive Access tier is currently not supported.
Encoding
The character set used to decode the contents of the file. Select Binary to read the file as binary and skip decoding. Select Auto to decode the file using the BOM of the file. Otherwise, use a specific encoding type to decode the file.
Include Metadata
Includes metadata specific to S3. Will include System-defined and User-defined object metadata.
Output Settings
Bucket Name
The name of the bucket to write output data to. This field supports dynamic output references.
Key
Specifies the default file name in S3. If left empty the file name is a GUID with a timestamp. This field supports dynamic output references to control the key depending on the output payload.
Payload Reference
When working with complex payloads, this setting uses dynamic outputs to specify the attribute that contains the file payload (e.g. {{this.filePayload}} ).
Note: If left blank, the entire payload is written out.
Metadata
Sets system defined name-value pairs included with the value sent to the S3 bucket. For example, users can set the Content-Type of the payload to different types such as application/json or text/csv.
Storage Class
Specify the storage class of the payloads being written out to S3.
Prefix Path with UTC Time
When enabled a time prefix is added to the key to logically separate files in S3 as yyyy/MM/dd/HH/key.