Meanwhile, users are left to figure out how to cut their cloth
A relative newcomer to the enterprise data and analytics world, Microsoft didn’t hold back when it launched its Fabric platform last month.
Against companies like SAS and Teradata – with more than 95 years of history between them – the Redmond software giant, which only launched its Synapse data warehouse in 2019, promised to address «every aspect of an organization’s analytics needs.»
It is a bold claim to make to organizations whose needs may already be being served by complex layers of vendors, technologies, and architectures, each serving different business needs or user populations.
Microsoft’s decision to jump in with both feet was foreshadowed by some moves by other big hitters in providing cloud-based data lakes, warehouses, and analytics.
In January last year, cloud-based data warehouse company Snowflake announced external table support for Apache Iceberg in private preview, followed by general availability in the summer. Cloudera followed suit in July, while Google announced its support for the open source table format in October last year.
All this matters because it promises change the economics of analytics, allowing users to bring analytics to the data rather than expend the money and effort moving data into a specific repository.
Now Microsoft is doing something similar, in a slightly different way. The company has announced its support for the table format Delta, which is open source via the Linux Foundation, but gets the majority of its contributions from Databricks, the AI and analytics company once famed for backing unified analytics engine Apache Spark. SAP also backed Delta through its partnership with Databricks, although both companies said they would support Iceberg and Hudi, another table format, in the fullness of time.
But Microsoft went with Delta owing to market demand, Arun Ulag, corporate vice president of Azure Data, told The Register.
«If you bring data into the data warehouse, it’s putting data in its own proprietary format, which from a customer perspective is not great because they feel locked-in: each time they touch their own data, they have to pay somebody to be able to do that. So, in Fabric that goes away. The native format for Fabric is the open source data format, which from a customer perspective has been really exciting because if it liberates the data, it allows them to use the entire ecosystem of open source tools against the data,» he said.
Although support for Iceberg and Hudi will be coming externally, Ulag explained that, by default, Microsoft Fabric would favor Delta and Apache Parquet, the column-oriented data file format.
«We have introduced in Fabric our native format, by default is Delta and Parquet,» he said. «It is a big deal because it’s not an external table. It’s not something that, if the data exists, you link to Fabric.
