AWS Lake Formation Reviews

We use AWS Lake Formation extensively. As a user, I operate through a consumer account where I utilize Athena, Glue jobs, and Step Functions. All the AWS services I create use AWS Lake Formation, though it remains abstract since it serves as the foundational layer structured by data engineers. Through the consumer account, I access data stored in the data mesh using AWS Lake Formation.

The governance system requires requesting access for databases, with AWS Lake Formation managing this governance. The data mesh functions through each business unit in our company creating data inside S3, called Source of Record (SOR). They maintain the schema on a Glue Catalog, administered by AWS Lake Formation. This structure enhances data integrity since the responsible area maintains it and creates layers that serve as a source of truth. They publish comprehensive data documentation and take responsibility for it.

Prior to AWS Lake Formation, we used a basic lake structure where raw and structured data were stored together without documentation requirements or clear responsibility for data storage and pipelines. AWS Lake Formation significantly improves the structure of the data mesh, making it superior to previous structures we used.

In my company, Itaú, we don't utilize all AWS offerings due to rigorous security measures. We operate approximately six to eight months behind other available services. I'm uncertain if gaps exist because of this limitation, though the system functions effectively for us. AWS Lake Formation offers column-level access control for databases, but we haven't implemented this feature either because it hasn't been approved by our compliance, governance, or security areas.

In our current setup, everyone from my business unit uses the same consumer account. When access is requested for a table, everyone using that business unit account receives access. This could present a security concern, though it benefits new team members who automatically receive all necessary access permissions. However, I struggle to identify specific improvements needed in AWS Lake Formation.

I have used the solution since the start of 2023.

The system is very stable. The only issue I encounter involves query timeouts, which I attribute to configuration settings rather than AWS Lake Formation itself. Queries exceeding five minutes or reading more than 10 gigabytes of data exceed resources and stop. Beyond this timeout issue, stability is excellent. I cannot recall any system outages affecting the data mesh, connections, or governance in the past two years.

Our experience with customer service is unique due to our company's size of approximately 100,000 employees in Brazil. The person responsible for Amazon contact, support, and maintaining AWS Lake Formation stability, including tables, connections, governance, roles, and user management, works in a separate department. When issues arise, I communicate through my management hierarchy, who then contact the responsible department rather than reaching out to Amazon directly.

Positive

AWS Lake Formation is a complex service, so I recommend studying and gaining substantial knowledge before implementation. It can be implemented over various services such as EMR or Hadoop, making thorough understanding crucial for successful application. My rating for AWS Lake Formation is 9 out of 10.

Private Cloud

Amazon Web Services (AWS)

My use cases include real-time healthcare analysis at Kaiser, where we build real-time streaming pipelines using Kafka and Apache Flink to process patient monitoring data. Data from electronic healthcare records is ingested through Kafka. It is processed in Flink with event time aggregation and stored in Delta Lakes for downstream analysis. We use this to reduce latency from minutes to seconds, as we aim for real-time visibility into patient healthcare monitoring.

The other use case involves using AWS and Azure together, specifically in stream and batch processing pipelines for IoT devices in drilling equipment at Kaiser, where we use Spark on Hadoop for large-scale data processing and Kafka and Azure Event Hubs for real-time ingestion. We also enable detection of models, such as predicting equipment failure, which reduces downtime by 20% and cuts infrastructure costs by 40% through migration to Azure.

The feature I find best is event time processing and checkpoint features that allow me to handle late-arriving events with accuracy while still guaranteeing exactly-once delivery, which is critical for healthcare data pipelines. This is what I appreciate about Apache Flink.

Regarding Kafka, high throughput and fault tolerance through partitioning and replicating stands out for me. This is the most significant plus point for the schema registry, which is very useful for keeping data consistent across production and customers.

Regarding technology such as Kubernetes, I appreciate auto-scaling and rolling updates, including scaling up and down automatically based on job needs during peak loads without downtime, which improves system reliability. With Azure Data Factory, I find the parameterization of pipelines and monitoring dashboards particularly useful.

I think AWS Lake Formation could improve by enforcing the least privilege by design, moving from ad hoc grants to role-based access controls.

In Kafka, using one search, such as index-level rolling, can be enhanced, and in Spark Synapse, applying row and column-level security along with dynamic data masking for PHIs, as implementing temporary audit access for on-call or break-glass scenarios, are improvements I envision.

8 on a 9 scale

The challenges I face involve managing state and checking in large-scale jobs, encountering long recovery times and failures, where we optimize intervals and tune state backends to significantly reduce recovery time. Additionally, I deal with message spiking during peak loads, which we solve by partitioning topics more efficiently, tuning intuition policies, and scaling customers with Kubernetes auto-scaling.

Another challenge I encounter is resource contention across different data pipelines, and I implement quotas and node pools for horizontal pod auto-scaling to balance workloads and prevent bottlenecks.

The support team assists me during on-call incident management, and I participate in on-call rotations when P1 or P2 tickets are raised for ServiceNow or PagerDuty. I help troubleshoot by analyzing logs, checking Flink and Kafka job statuses, resolving Kubernetes spot failures, and continuously improving upon recurring tickets while assisting the team in identifying patterns.

Positive

We are customers of Amazon and use it as a product. At Apple, we utilize S3 for storage and Redshift for analysis, along with CloudWatch for monitoring, IAM, and security access management. The overall rating for this solution is 8 out of 10.

Public Cloud

Amazon Web Services (AWS)

We had use cases for AWS Lake Formation, specifically for a government client that was trying to migrate a lot of data and implement a solution that enables various entities to access the data at a central location, since the government entity had data coming in from different departments across various government entities. They wanted to build a data lake that incorporated big data processing with Glue services and crawlers, while also proving out a concept for securing their data lake and data assets along with providing governance around it, allowing different entities to access the data under secure conditions with scalability in mind. I was a consultant for one of the entities down here, where I was involved in a sort of POC to see if they could potentially use AWS Lake Formation along with Security Lake.

The most valuable features of AWS Lake Formation were the access model itself, as it allows implementation of filters, Blueprints, and row-level and column-level security to mask data that shouldn't be accessed by certain entities, enabling granular control without exposing PII data. Another feature is the Glue Workflows, which allow orchestration of multiple Glue jobs to automate the entire process end-to-end. Additionally, the Blueprints feature, which provides connectors out of the box for ingesting data from different sources, was also beneficial. AWS Lake Formation is tightly integrated with IAM for authentication and authorization, as its permission model relies on IAM user groups and roles. This allows categorization of groups based on the access required by different users, enabling implementation of access policies within Lake Formation. This integration provides extensibility and scalability since user groups, once granted permissions, can manage further access control for new users or groups. While we were still exploring other features such as federated access for users outside AWS, we were in the early days of utilizing AWS Lake Formation. The scalability of AWS Lake Formation is quite good, allowing creation of user groups with grantable permissions, letting users manage access for new users onboarded to specific databases or tables, as these groups can grant permissions to extended users as needed. The stability and reliability of AWS Lake Formation are impressive; once permissions are applied, the access flow is efficient. When a user runs a query from Athena, it interacts with AWS Lake Formation first, which uses temporary credentials to access S3 buckets and presents data securely. This centralized permission management adds a layer of security, making it predictable in what users can access while applying necessary filters before data exposure.

In terms of improvement areas for AWS Lake Formation, one feature I noticed about 10 months ago is the ability to query data from Athena without accessing S3 buckets directly; users granted access to tables can run SQL queries via Athena. However, I found that Athena can be a bit clunky when writing queries, indicating a potential enhancement point for easier user interaction with query tools such as DataGrip using provided driver JARs.

I've been working with AWS Lake Formation for about a year, specifically for about three to six months on one of the projects related to Lake Formation.

While I was not directly involved in the actual deployment of AWS Lake Formation, I found the application perspective to be straightforward for understanding features, although there were conceptual aspects users need to grasp before implementing permissions. There was some trial and error, but over a period of three to six months, we effectively scaled the permission models and controlled access to S3 files, while my colleagues faced challenges with securing S3 files and managing policies such as ACLs, ACLEs, and NACLs during implementation.

The stability and reliability of AWS Lake Formation are impressive; once permissions are applied, the access flow is efficient. When a user runs a query from Athena, it interacts with AWS Lake Formation first, which uses temporary credentials to access S3 buckets and presents data securely.

The scalability of AWS Lake Formation is quite good, allowing creation of user groups with grantable permissions, letting users manage access for new users onboarded to specific databases or tables, as these groups can grant permissions to extended users as needed.

I participated in the initial setup and deployment of AWS Lake Formation primarily from a Glue side, focusing on data engineering aspects, building Glue jobs, and creating S3 buckets while implementing permissions to different user groups.

I participated in the initial setup and deployment of AWS Lake Formation primarily from a Glue side, focusing on data engineering aspects, building Glue jobs, and creating S3 buckets while implementing permissions to different user groups.

For fine-grained access control of data assets in AWS Lake Formation, I used LF-Tags to categorize certain databases and tables based on tags assigned to specific user groups, allowing me to match tags and grant access accordingly. I also set Data Filters to mask either rows or columns on the tables and databases, and implemented S3 policies to restrict access to the S3 buckets. This involved creating groups and assigning permissions such as grant access, select access, update access, while these database and table levels were populated using Glue crawlers. I did not use AWS Lake Formation's machine learning algorithms for data classification, as the use case was focused on managing large volumes of data, which was close to 1.5 petabytes, rather than implementing AI models. On a scale of 1-10, I rate AWS Lake Formation a 7.

Public Cloud

Amazon Web Services (AWS)

We have been working with AWS Lake Formation while working with multiple companies and enterprises. I have worked with AWS Lake Formation for three or four years with multiple AWS Lake Formation groups. We used AWS Lake Formation last year in 2024, during August and September. We set it up for an enterprise client in India.

In the shortest form, what I appreciated about AWS Lake Formation was that the schema definition and data cataloging were quite good. The Glue Data Catalog, which was used for tables, conversations, usage stats, and customer profiles that we built out, was good. We created the S3 and put it as a central data lake out there, which helped us significantly. Moreover, the best part was the fine-grained policies that we used. For example, the policy-aware retrieval we implemented when we had to show competitor data allowed specific groups of users to access particular data sets.

We utilized the fine-grained access control for data assets in multiple places. For row-level access control, we gave store-level data only to regional managers. For column-level control, personal information and phone numbers were hidden from analysts but given to customer support. We set up centralized policies that reduced the usage of IAM access management and S3 usage because the centralized policies helped us do that automatically for each user and department within the industry. Using CloudTrail, we set up an auditable trail for multiple people, which allowed us to see which data was accessed by whom. We could set up analytics on Spark using that.

I would appreciate improvements in the onboarding process within AWS Lake Formation. The ease of setting up policy management needs enhancement. Currently, configuring it manually takes considerable effort from the organization and requires collaboration between teams to determine the right policy configurations. Additionally, rows and columns are the only control style available. I would prefer to have time-bound policies so that these policies are active only for a particular time for a specific user. This would have helped during situations when business analysts were temporarily working for our organization, as we had to manually cancel out the policies.

AWS Lake Formation has improved collaboration in the form of governance. Previously, data was kept on S3, and manual settings needed to be created. We had to set up many things independently, create IAM users, and create policies for everything. Governance became easier, which made giving access to different data teams more straightforward. Access to data became centralized yet controlled through AWS Lake Formation.

I have worked with AWS Lake Formation for three or four years.

I assess the integration of AWS Lake Formation with IAM solution for authentication and authorization as difficult in the beginning because it requires significant thought, and large enterprises needing to shift to it will require extensive compliance. The initial onboarding process is challenging because creating a plan takes a month to a month and a half to build out. After that, integration and deployment begin. It becomes easier to use afterward, though updating sometimes presents challenges. Overall, the long-term gains are worthwhile.

We had a positive example with company-wide data, user data, and user chat histories. The data was segregated easily, and data control was established. Customer support required PII information while the business analytics team required different data sets, which helped improve collaborations on common databases. The negative aspect was that due to governance requirements, we observed an increase in latency in the query workload. The P90 increased by around 5-6%, but this was manageable for internal workflows and customer support.

We had business support from AWS technical support and customer service, and they provided significant assistance when we needed help with integration. AWS is friendly in this regard, and they are quite fast and efficient. On a scale of one to ten, I would rate the tech support of AWS as an eight.

Positive

Apart from AWS Lake Formation, I have worked with multiple competitors and similar solutions. We tried using Snowflake and studied Databricks, the lake house platform built on Delta Lake. While Databricks' ML pipelines are strong, we didn't need the ML/AI pipelines. Snowflake's governance and sharing capabilities were better than AWS Lake Formation. However, since our client was already deeply integrated with AWS, AWS Lake Formation made the most sense.

ROI on AWS Lake Formation wasn't our primary focus. Our success criteria were based on ease of governance and management of data usage rather than cost optimization. We were very successful in these areas. Previously, there was no trail of who was accessing what data, but CloudTrail helped us significantly in this regard.

The costing of AWS Lake Formation is reasonable. It depends on which organization is trying to use it. The organization we worked with was comfortable with the cost because they were facing major data policy issues. AWS costs are usually very competitive.

Given my extensive experience with AWS Lake Formation, I would advise any organization considering it that when working with data lakes and data governance, thorough planning with multi-stakeholder involvement is crucial. If planning is done correctly, governance will be set up properly regardless of the platform chosen, whether it's AWS, Azure, or Snowflake. Each stakeholder's requirements should be gathered, and clarity should be provided on data access permissions.

The onboarding process typically takes one to one and a half months for planning because in large enterprises, multi-team collaborations can be challenging. I recommend taking that time to build a comprehensive plan before starting implementation.

Based on my experience, I rate AWS Lake Formation 8 out of 10.

Amazon Web Services (AWS)

AWS Lake Formation helps me build my infrastructure for managing data lakes. I can bind it to an S3 bucket and then manage it.

I use AWS Lake Formation because my stack is in AWS and I use it for managing my infrastructure as code.

For AWS Lake Formation, I'm using Terraform, and the support with Terraform is very good.

I'm using the administrator access for AWS Lake Formation for building my resource policies while getting used to it.

I have integrated AWS Lake Formation with the IAM solution.

The integration of AWS Lake Formation with the IAM for authentication and authorization is very good. I didn't have any problems in the setup. I thought it was simple. I've attached my user to a group in IAM and then linked it to my administrator access policy in AWS Lake Formation via Terraform.

I have been working with the AWS Lake Formation solution for over six months.

The implementation was smooth as far as I understand.

One important step is to evaluate your requirements, your non-functional requirements. In my case, using a SQL Server is not very good, so a tool AWS Lake Formation can be very useful. You are going to decouple your online transaction operation from your analytical one. AWS Lake Formation is very useful to manage it and to do access control.

The FinOps team holds the information about the pricing, setup cost, and licensing cost of AWS Lake Formation.

I haven't seen any measurable benefits from using AWS Lake Formation, such as time saving, resource saving, or efficiency improvements.

In the management of our access control, I didn't have any performance issues with AWS Lake Formation. While using Athena for querying my data in my data lake, I had a high cost and latency that was a problem in the beginning of the project. When we moved the solution, it improved.

AWS Lake Formation is not as simple as Databricks. I'm getting started with it, but I won't have very many contributions.

An easy tool for querying my data in AWS Lake Formation at scale at a lower cost can improve my experience and make it a 10 out of 10.

I have been working with the AWS Lake Formation solution for over six months.

For AWS Lake Formation, I'm using Terraform, and the support with Terraform is very good.

I have integrated AWS Lake Formation with the IAM solution.

The integration of AWS Lake Formation with the IAM for authentication and authorization is very good. I didn't have any problems in the setup. I thought it was simple. I've attached my user to a group in IAM and then linked it to my administrator access policy in AWS Lake Formation via Terraform.

The implementation was smooth as far as I understand.

My company has a thread in Slack where we can contact AWS technical support.

I had one call with them in the beginning of the project, about two months ago.

I just had it one time, but I would rate it an eight or nine, maybe nine.

Positive

In the past, we generated reports in a SQL Server database. It was very critical for OLTP. We are modernizing our architecture and using an OLAP DB. Our S3 serves as a data lake. AWS Lake Formation in this context was a change of our architecture. We're comparing it to other solutions. We're using a Delta Lake in S3, but we're also studying another tool such as Databricks.

For getting started in Terraform, I required a provider from AWS HashiCorp. Then I set policies and the user for IAM. Then I linked it to my S3 bucket. I was very used to doing that for another project so that step was simple. In data lake, I've done the location. I've assigned my ARN from my S3 bucket. It was very simple, and the Glue catalog. That was a first step from our setup that I remember.

We don't have metrics on how AWS Lake Formation supported the data analysis processes for me, but we could discuss this at another time when I collect them.

One thing I'm studying about Databricks that differs from what we're using in AWS Lake Formation is the easy way to set up the environment and accept a Delta structure. It's very simple in comparison.

I'm using Snowflake and getting used to the architecture of Snowflake for horizontal databases. It would be a good shape, a lakehouse, a simple tool in AWS.

We're initiating the use of AWS Lake Formation in our company. We are just starting talking about the tool. Not many teams use it, but we're getting started and it has been useful in production.

We were using AWS Lake Formation for managing our data lake, but we also had Athena. Athena was very expensive in the beginning. We had to find another strategy for economy purposes.

My overall rating for AWS Lake Formation is 8.5 out of 10.

Public Cloud

Amazon Web Services (AWS)

In my experience, I have mostly worked with AWS, but I did have a little experience with Google Cloud, although I didn't develop on Google Cloud; I had some limited contact.

In my experience, I have worked primarily with AWS Lake Formation for data warehousing, as I had a significant contract with AWS while working in my last corporation, Natura Pay, followed by my time at Itaú Bank, which had a major contract with Amazon, making AWS the dominant experience for me.

The best features in AWS Lake Formation are the logs in real-time. CloudWatch is very good because it is easy to learn and understand the type of the real.

The main benefits that I have seen from using AWS Lake Formation are related to FinOps because you have control of your data and can track your costs since AWS Lake Formation is integrated into a unique platform, which is AWS Cloud Service. You can see the integration of your deployment when you develop a new table using your deployment platform integrated with the Cloud such as GitHub, making it easy to control your logs, optimize your FinOps, and share information with all people in the organization.

I recommend AWS Lake Formation because it optimizes the work of people in general, making it easier for everyone.

I don't know any features that I would want to see included in the future releases of AWS Lake Formation.

Making queries in SQL is a reality now, and in the future, it would be beneficial if AWS implements a feature where AI can provide better analytics, because nowadays AI can produce good analytics, but it's not ideal. If AWS introduces better analytics features in the future, I think it will be very beneficial.

I am uncertain if I have used AWS Lake Formation's feature for collaboration inside my data teams.

AWS Lake Formation is growing in importance because corporations nowadays can buy more physical computers and it is necessary to purchase more cloud services. The future is critical because data is extremely important for making decisions and optimizing the corporation's money.

Finding answers to questions about AWS is easier through research on the internet, as there is a lot of information and a community using the service. I find a lot of information about problems online; however, I would appreciate online support, which I don't have access to in my corporation at the bank, so that is important.

Positive

I prefer AWS over Google, but I recognize that I don't know a lot about Google. I find AWS logs more user-friendly; in the past, they have been extremely organized, whereas Google Cloud was harder for me to understand the information.

The setup for AWS Lake Formation involves opening a repository. My first activity is to create a repository for the infrastructure and create a schema in Terraform. After that, I create the Terraform structure with the necessary information such as columns and their types, which is extremely important, and I make the deployment because this creates the table in the AWS platform. I then create a new repository for my Glue job to consult the table and organize the permissions, rules, and all information in Terraform for the initial setup to connect my job with the platform.

The initial setup of AWS Lake Formation is easy; you might find it challenging at first because you may not know the errors or challenges, but once you've done it once, the subsequent times become easier as the problems tend to be similar.

Users considering AWS Lake Formation need to understand that it requires an investment. You must consider that your corporation will input a lot of money into the services, but in the future, it will optimize your costs and speed up your work with data. It is a service that enables your corporation to drive data-driven decisions, which is extremely important, so I think this investment is necessary.

I don't understand much about the pricing of AWS Lake Formation, but I know how to search for the cost of Glue jobs, and I use the calculator in Amazon. I use a tool to preview the cost based on the workers and processing of my Glue jobs, and AWS has many services to control your FinOps, including the calculator.

I am uncertain whether the pricing of AWS Lake Formation is reasonable or if it is too high.

I develop ETL when information is in the Serverless environment on the physical computer; you change the space and democratization, but I don't work with machine learning.

My review rating for AWS Lake Formation is 10 out of 10.

Public Cloud

Amazon Web Services (AWS)

Most of our use cases with AWS Lake Formation were around handling the permissions for most of the tables and databases that were in Athena and Glue. Our main use case was using AWS Lake Formation for permissions and handling the users around the Glue databases and tables.

A favorite feature of AWS Lake Formation is that it provides us with visibility into who has access to a particular table or database in Glue. That is the kind of feature I appreciated most.

The main challenge we faced with AWS Lake Formation was related to cross-account sharing. Granting access to other AWS accounts for tables or databases in a different AWS account was somewhat difficult. They might have introduced something new or updated the features since then, but cross-account sharing stands out as a drawback in AWS Lake Formation.

I have used AWS Lake Formation for two to three years. I am not using it in my current role, but in previous roles, I used it actively.

There was no maintenance needed from our end for AWS Lake Formation.

There were no crashes or stability issues.

In terms of scalability for AWS Lake Formation, it performs well. Once you set up your data lake, if you keep creating tables or databases inside the data lake, they are automatically present in AWS Lake Formation. We never worried about scalability for AWS Lake Formation and never had any issues in that regard.

I am just a customer.

Neutral

Most of my work has been with AWS, so that is the only tool I have used. I have not used any alternatives.

The initial setup is not extensive. When we were creating Athena tables from S3, it took approximately an hour. It is not difficult to set up.

It was implemented by just one person.

When we started using AWS Lake Formation, we initially used it for granting permissions and access to users through resource-based permissions. Instead of resource-based permissions, new users can start using LF-Tags, which is more robust and recommended by AWS. That is the suggestion I would pass on if somebody starts using AWS Lake Formation anew. On a scale of 1-10, I rate this solution a 7.

Amazon Web Services (AWS)

I was a consultant building a solution for a customer who had data scientists working across different business areas. The data science team needed a reliable and trustworthy platform where they could create and manage their own data structures independently. The project involved developing a foundation and structure to enable their work. Since the data scientists were scattered across business areas, each area required its own space to work with data, develop, and test. The solution delivered was a data lake managed by AWS Lake Formation, which allowed control of permissions, visibility, and governance at the business area level. Each business area had its own space where internal members could access data, while other business areas could not access it unless permitted.

Early in the project, business areas requested a shared space to share data with other departments. We delivered a solution where each business area was controlled by LF-Tags inside AWS Lake Formation, where resources were tagged with specific private or public sharing tags, and resources followed the rules attached to the LF-Tags.

The LF-Tag system itself allows for very granular control of permissions. Once the starting permission scheme is defined for each business area, they can control it themselves without risk or management overhead from IT. The LF-Tag system with granular permissions was key to the project as a functionality of AWS Lake Formation.

It served as the cornerstone of their governance strategy. The project would have taken much longer or been more complex without AWS Lake Formation. The project's main goal was enabling business areas to collaborate while maintaining private areas, making AWS Lake Formation central to the customer's solution.

The system proved quite challenging to configure correctly. Our architecture used EMR for compute and data transformations. Business areas would spin up clusters from a custom website rather than having access to the AWS management console. Through this website, they could choose predefined cluster types and storage, create clusters, and connect to use Apache Spark for computing and data transformation.

The architecture involved multiple components, including a Sagemaker notebook for users to connect to Spark, with Spark directly accessing data. The nodes in the Spark cluster needed to communicate with AWS Lake Formation to access data. We encountered issues with forwarding user identities from the front end to the backend where AWS Lake Formation resided. Despite following official documentation, configuration problems persisted, requiring weeks of support from multiple AWS engineers to resolve.

The user interface in the console of AWS Lake Formation is cluttered, making it difficult to find specific options. The permissions section is particularly confusing with multiple paths to reach the same destination, making navigation ambiguous.

I am not currently using this solution.

The solution was highly available. Initially, we had to list all business areas and their tags, which was necessary regardless of the tool used for governance and data lake management. Regarding scalability, we encountered no problems at all. It proved highly scalable and straightforward to implement.

The initial setup was straightforward using Terraform, an infrastructure as code tool. We coded all configuration options and object creation through this tool. There were minor issues during implementation, and since it was a relatively new tool at the time, the learning curve was steeper than with other tools.

The support experience was inadequate. After creating a ticket, it remained unresolved for weeks. AWS frequently changed support personnel, with slow response times often taking days. Each new person assigned to the case lacked context and repeated questions that had already been answered. The situation became serious enough to require escalation to the customer and nearly impacted the project delivery schedule.

Positive

Public Cloud

Amazon Web Services (AWS)

We are the review platform and informational platform that works with different vendors to collect thoughts and opinions on products. We have partnered with AWS and are helping them collect feedback regarding AWS Lake Formation solutions.

In my recent projects, I have over seven years of experience working as an AWS data engineer. I have used all AWS services throughout my projects according to requirements. I have used AWS Glue to build robust pipelines and Lambda to trigger the pipelines when an event occurs. I have also used Apache Airflow to automate all pipelines. Additionally, I have utilized SNS (Simple Notification Services), Amazon QuickSight, and Oozie. All these services have been implemented in my project according to their specific uses.

AWS Lake Formation has the ability to automatically classify and enforce security policies on the data.

There are areas that could be improved in AWS Lake Formation, and functionality that could be included in future releases.

I have been using the solution for two years.

The initial setup of AWS Lake Formation had some challenges.

There have been improvements with the speed of deploying BI tools and ML applications with AWS Lake Formation.

The technical support from Amazon has been discussed.

Positive

The initial setup of AWS Lake Formation presented some challenges.

We have observed measurable benefits in terms of cost savings, time savings, resource savings, and efficiency improvements in our workflows.

The pricing structure of AWS Lake Formation has been considered.

I have experience working with cloud databases and data lake storages from other vendors. When comparing AWS with Azure and Snowflake, there are distinct advantages and disadvantages to consider with each platform.

I have recent experience with AWS Lake Formation and have worked with its centralized repository. The integration experience with S3, Glue services, and IAM has been noteworthy.

The overall rating for AWS Lake Formation is 4 out of 10.

Amazon Web Services (AWS)

My usual use cases for AWS Lake Formation involved securing and governing the data resources that we configured in AWS, but we did not use the analytics or machine learning capabilities specifically. For example, with the metadata in AWS Glue, we secured and governed that. For analytics output and Step Functions, we configured some permissions with AWS Lake Formation, but the roles themselves provided the general policy and interaction with other services.

We did not use it to share data internally or externally in terms of service across accounts, which made sense because it was specifically for a client where there was no interest in that particular solution.

Because of that, the orchestration and the way we used it was very limited. In fact, we rarely interacted with it because the main interaction, when configuring it for this generic type of form when solutionizing for a client, would be configuring and handling varying types of operations for RDS, creating a pool in Cognito, working more in SageMaker, configuring step functions with Lambda functions, and similar tasks.

The features and capabilities of AWS Lake Formation that I have found most valuable are that it is really convenient to see all the different data assets that were configured and understand who has and what type of service has or does not have access to those services.

Additionally, seeing which type of IAM roles or IAM in general are related to that type of configuration allows us to explore and error handle when we need to figure out if some policy was missing.

From a governance perspective, it adds significant value to figure out these types of things, though I must say there are better services in AWS overall for governance.

Regarding areas of AWS Lake Formation that could be improved or enhanced, I prefer not to answer, mainly because I do not believe that I would be the most valuable person to ask, as I have not used all of its different capabilities to the maximum to give an opinion.

Regarding optimization of the governance aspect of AWS Lake Formation in the future, people in general would prefer not to work with multiple tools that do nearly the same thing. I would rather have a tool that is specialized for governance and that is its sole purpose. Rather than creating an additional hundred tools, optimizing a tool to have a centralized location to do governance would be beneficial. Then users would not need to search extensively to figure out varying capabilities. Of course, there are some edge cases such as AI, where governance and guardrails might require different solutions. The general scope or principle is if you incorporate IoT or a lake house or do machine learning orchestration, you would rather simplify the scope instead of complicating it with many different types of services.

I have worked with AWS Lake Formation previously, and it was almost twelve months ago, but actually more six months ago.

AWS Lake Formation is very stable. I have never encountered any issues or locations or availability zones that went down. They are all very stable solutions.

AWS Lake Formation is very scalable for the use cases that I worked with, highly scalable for that matter. Because we used it for a very specific purpose, its full scalability capabilities were not fully tested. Since the orchestration of the solution is configured upon varying types of services and not AWS Lake Formation as a central location, I do not believe that in that orchestration it is really significant.

I have utilized AWS Lake Formation's fine-grained access control for data assets. Some data assets we have allowed access to, while others we have not configured. It depends on how you structure your solution. We have not used AWS Lake Formation for all the different types of data assets because varying types of clients require varying types of security considerations. It is mainly used with RDS Athena.

I was a customer of this service, and the company I worked for used it for the company itself. My interaction with it was creating a solution for a client.

I really love AWS and when compared with Azure, it is a bit more complicated overall, but AWS is significantly more robust, and you learn to appreciate that when you work with it and see how scalable it is. Maybe the UI is not entirely the best in the world, but it is very clear in its structure and there is always documentation that is really good that you can read about and figure things out.

This review rates AWS Lake Formation 5 out of 5.

Amazon Web Services (AWS)