We are also using Apigee we have various consumption patterns, data enrichment, and few shedding of the data, and everything goes into Snowflake. If it is multiple consumers, it goes into AMQ, Kafka, or multiple streams to consume. There are specific APIs that we offer after we send the data into the S3 bucket. We have Apigee APIs for consumption, and there are three to four different patterns. For example, we enrich the data, flatten it, and structure everything before the customers going to go into Snowflake.
There are going to be specific clients who need specific data from the overall data lake, those are going to be exposed as APIs. We have multiple customers needing the same data and for this, we move them into the streaming Kafka.
Apigee does not communicate directly with Snowflake. We have data registration, and everything is coming into something that is called the trusted bucket. The Apigee interface API is written off the S3 bucket. The S3 bucket data is moved into the Delta Lake, and where the data are stored from the Delta Lake, it sends it to Snowflake. We have Apigee going to Delta Lake and S3 bucket, but Apigee does not go to Snowflake, these are two areas where it goes to.
We have Kafka consuming directly off Delta Lake, and it sends data to Kafka through the AMQ. We have its setup, and we have interfaces that come directly to Snowflake to pull the data. It is then flattened and enriched, and it is used for many purposes, such as reporting.
Very good review on Snowflake, very helpful.