What is our primary use case?
I am familiar with Elastic Search to a certain extent as I have used it in my development life. I thought someone wanted feedback about it, specifically how I have used it in my career, so I agreed to share that information.
I started using Elastic Search after becoming acquainted with it when I accessed the AWS environment for the first time during the COVID period. We tried to establish a vertex and edge graph database schema, and I was hired to get that schema up and running while dealing with millions of records related to car spare parts. Due to a signed clause, I cannot go into too much detail. The challenge was with the indexes slowing down, which prompted a move to GraphDB because it provides faster access time. I had to deal with a lot of data cleansing and created many pipelines, first pushing records into Elastic Search through a bulk insert. I also looked up data using Kibana as the front end to leverage queries for pulling up that data.
Once GraphDB was in place, I was required to develop a service for asynchronous processing and order confirmation, where one copy would be stored in a database and the other would be pushed into Elastic Search for further lookup, eliminating the need for direct queries to the RDS
I have never reached out to Elastic Search's technical support team.
What is most valuable?
Elastic Search, being a vector database, quickly indexes data, allowing for searches based on text and data directly, which I found fascinating. My dev lead mentioned that it uses C++ to pick up these indexes and pulls up records incredibly fast, in nanoseconds, keeping me interested in how things are becoming faster over time and diversifying away from traditional relational database systems.
Regarding scalability, I consider both vertical and horizontal scalability in theory. I have not experienced sharding but find it interesting as a use case with Elastic Search. I see significant potential for vertical scalability, which can accommodate more data and offer substantial improvement.
What needs improvement?
Your question about what I dislike about Elastic Search is quite pointed, and I prefer to look at it as something for improvement, such as provisioning options other than Kibana. A standalone install that is operating system agnostic could run on Mac, Linux, or Windows by just providing a URL, username, and password to access the schema for queries. This would benefit many people who may not have access to Kibana, especially those who, the workplace evolution has shown, may not know what Kibana is if they lack tool access. It is crucial to have executable information to understand a product deeply. If Kibana is not a viable option for everyone due to hosting constraints, a standalone installer could connect directly to Elastic Search, with documentation readily available online to guide those needing desktop access.
For how long have I used the solution?
I have been using this solution for two years overall and have had good exposure to it with all CRUD operations I have been performing with it.
What do I think about the stability of the solution?
I have used Elastic Search for log lookups with ELK and never encountered any crashes or downtime while it was hosted in the cloud. While occasionally one or two queries may take longer due to network lags, these issues are more infrastructure-related since I have never faced any problems with Elastic Search's stability, which generally retrieves information instantly.
What do I think about the scalability of the solution?
Regarding scalability, I consider both vertical and horizontal scalability in theory. I have not experienced sharding but find it interesting as a use case with Elastic Search. I see significant potential for vertical scalability, which can accommodate more data and offer substantial improvement.
How was the initial setup?
When discussing initial deployment, the specific attribute of interest is the overall initial installation when starting to roll out the product. The deployment was a struggle as I faced challenges with bash commands and understanding how to run things on my system. Looking up tutorials on YouTube was tricky, and cross-referencing with documentation posed difficulties as some people customize setups to their needs. Setting up MySQL is straightforward, while with Elastic Search, I had to run bash commands for proper service execution. I faced some hurdles getting CRUD queries to work correctly. I resorted to Docker as an alternative, which diverged from standard practices of creating a local database service. An ideal setup would include a setup executable for Windows that would greatly facilitate immediate access and CRUD operation starts.
In my case, the system was already running by the time I started, as the custom DevOps team managed the deployment, and I was only tasked with connecting via Kibana and issuing bulk insert commands.
What's my experience with pricing, setup cost, and licensing?
I have not checked Elastic Search's pricing thoroughly, so I do not know how a company would perceive it. From what I see, small companies might consider the cost, with starting pricing for a single node instance at $16 a month for serverless and hosted options, though at least one or two connected clusters would be necessary for viable solutions. Companies might see this lower end pricing as suitable, but for startups, reaching up to $2,000 could appear steep, depending on their aggressive usage approach.
I faced a situation where our graph database work halted due to technical difficulties with the Neptune product, as some CRUD operations were not carried out. Product specialists suggested that the business case did not fit the graph database's requirements and recommended Elastic Search instead for a better use case. I was involved in a data structure related to car spare parts needing to facilitate purchases by linking parts to various car makes across catalogs, ultimately attempting to shift from relational databases due to overwhelming data generation that slowed down indexed lookups. Elastic Search significantly helped in confirming order data lookup, but costs for clusters in further development led to work being stalled.
A preliminary architect consultation or proof of concept on cluster purposes would aid in establishing understanding for further development on Elastic Search, which is becoming increasingly costly in the cloud due to demand. A structured understanding of costs tied to usage metrics would greatly assist in planning before commitments, as delays in our POC adversely affected our progress. Documentation should also encompass potential use cases and scenarios to better assist developers during implementations across programming languages to ensure seamless integration.
Which other solutions did I evaluate?
Regarding alternatives, I have worked with various database products, including Azure technologies where I worked with NoSQL storage tables similar to AWS DynamoDB, which are schema-less with varying attributes per record. These use partition key and row key for accessing information, fragmenting what we associate with traditional RDS. Additionally, I worked with Axelor CRM from a French company, alongside MySQL and Oracle. My first company used MS SQL, and I have discussed my use case involving AWS Neptune graph database and Elastic Search, which encompasses all I have worked with so far.
What other advice do I have?
For an overall rating of Elastic Search, I would score it at a solid 8 out of 10.
Its speed has facilitated my understanding of logical operators and streamlined query issuance. I would love to grasp the inner workings of sharding with distributed schema implications. Based on what I have experienced thus far, I find it a significant improvement, but once I better understand sharding and its performance effects, I would likely adjust my score.
My experience with the relevancy of search results using Elastic Search indicates that issuing a full query yields a finite number of results, while partial text searches can return irrelevant information. Mastering query issuance with Elastic Search is a valuable skill that develops over time. I prefer a structured JSON approach, utilizing properly sequenced clauses, which allow drilling down to a limited set of records that directly relate to the search context.
On hybrid search effectiveness, I think that AI is progressively offering more concise information. Providing more relevant keywords allows Elastic Search to generate results faster than other databases, such as RDS. The ability to engage with text directly simplifies understanding records and has a significant impact on AI functionality in rendering accurate results based on user needs.
Which deployment model are you using for this solution?
Public Cloud
If public cloud, private cloud, or hybrid cloud, which cloud provider do you use?
Amazon Web Services (AWS)
Disclosure: My company does not have a business relationship with this vendor other than being a customer.