Read this post about how having several products in a Data Platform increases solution complexity, but also gives flexibility.
Join the DZone community and get the full member experience. I’ve been working as Solution Architect for many years, and I’ve seen the same mistakes very often. Usually, companies want to evolve their data platform because the current solution doesn’t cover their needs, which is a good reason. But many times they start from the wrong starting point: I think the reasons are simple, and at the same time are very difficult to change: Making mistakes and identifying them is all part of the learning process. It is a good indicator that we are working to improve and evolve our solutions. But we have to analyze deeply and understand the reasons (what, why, etc.) to avoid making the same mistakes every time. There are no magical data products, or at least, I don’t know them. A global Data Platform has many use cases, and not all of them can be resolved by the same data product. I remember when RDBMS Databases were the solution for all use cases. So many companies would invest a lot of money and effort in very big RDMBS (OLTP/OLAP) Databases as Teradata, Oracle Exadata, IBM or DB2. After those, came the Big Data ecosystem and NoSQL, and again everybody began to design Big Data Lake even as Data Warehouse using solutions such as Impala and Hive. Many of these solutions failed because they were only technological changes following the trends of the moment, but almost nobody had analyzed the causes: Today, we are in a more complex scenario. There are more data products (and marketing) joined with a new cloud approach. Cloud requires, more than ever, to change our culture, vision, and methodology. Having several products in our Data Platform increases the solution complexity in many terms such as operations, coexistence, or integrations, but at the same time gives us the flexibility to provide different qualities and also avoid locking vendor scenarios. We have to take advantage of the benefits of the cloud. One of them is to reduce operational costs and effort that allows providing a better solution in terms of the variety of services. This architecture provides a single layer that enables the users and applications access to the data uncouple the location of the data and the specific technology repository product. This approach improves the data integration experience for users by allowing us to evolve the data repositories layer without impacting the operation of the systems.
