OCI Streaming Service – Archive to Object Storage

To archive streaming data to object storage:

#1 Create a bucket

#2 Create a service connector. The streampool must have been defined ‘public’. Use a batchtime of 60s to ease the testing

#3 Publish a message

export COMPARTMENT_OCID=ocid1.compartment.oc1..xxx
export STREAM_POOL_OCID=ocid1.streampool.oc1.eu-frankfurt-1.xxx
export STREAM_OCID=ocid1.stream.oc1.eu-frankfurt-1.xxx
export ENDPOINT=https://cell-1.streaming.eu-frankfurt-1.oci.oraclecloud.com

oci streaming stream message put --stream-id $STREAM_OCID --messages '[ { "key": "a2V5MQ==", "value": "dmFsdWUx" }, { "key": "a2V5Mg==", "value": "dmFsdWUy" } ]' --endpoint $ENDPOINT

#4 Check the bucket

After 60s, which is the batchtime specified, the message will be written into the bucket, in their original format. Check the file once downloaded:

% cat ocid1.serviceconnector.oc1.eu-frankfurt-1.xxx.0.data

{"stream":"apps-stream","partition":"0","key":"a2V5MQ==","value":"dmFsdWUx","offset":12,"timestamp":1697187496586}
{"stream":"apps-stream","partition":"0","key":"a2V5Mg==","value":"dmFsdWUy","offset":13,"timestamp":1697187496586}
{"stream":"apps-stream","partition":"0","key":"a2V5MQ==","value":"dmFsdWUx","offset":14,"timestamp":1697187498455}
{"stream":"apps-stream","partition":"0","key":"a2V5Mg==","value":"dmFsdWUy","offset":15,"timestamp":1697187498455}
{"stream":"apps-stream","partition":"0","key":"a2V5MQ==","value":"dmFsdWUx","offset":16,"timestamp":1697187499640}
{"stream":"apps-stream","partition":"0","key":"a2V5Mg==","value":"dmFsdWUy","offset":17,"timestamp":1697187499640}%

References

https://docs.oracle.com/en-us/iaas/Content/Streaming/Concepts/streamingoverview.htm