New features focus on further improvement of S3 API functionality and data management with security, high availability, and scalability.
Alluxio, the developer of the open-source data orchestration platform for data-driven workloads such as large-scale analytics and AI/ML, has announced the availability of version 2.8 of its Data Orchestration Platform.
This new release features enhanced interface support for the Amazon S3 REST API; security improvements for sensitive applications with strict encryption compliance and regulatory requirements. It strengthens automated data movement functionality across heterogeneous storage systems without the need to manually move or copy the data.
Why It Matters
Alluxio 2.8 enhances S3 API compatibility, so onboarding and managing Alluxio on existing large data pipelines becomes easier. As a major enhancement to enterprise-grade security, Alluxio 2.8 adds features to support server-side encryption capabilities for securing and governing data.
Data migration is a major challenge for organizations having data stored cross-vendors, cross-clouds, or cross-regions. Alluxio 2.8 improves the reliability of data movement across heterogeneous storage systems with enhanced usability of policy-based data management and high availability.
Problems Solved
According to Adit Madan, Director, Product Management at Alluxio, the S3 API allows a myriad of applications, in languages such as Python and Golang, to access data via Alluxio without the need for a custom Alluxio client driver. Eliminating the need for a custom client expands the applicability of Alluxio to multiple use cases served by a data platform.
The goal of the security enhancements is to limit the data plane storing sensitive access credentials in a highly connected environment, especially for enterprises transitioning from an air-gapped on-prem environment to a hybrid cloud.
“The support for S3 API is significant, making Alluxio much more accessible to a huge number of customers,” said Adit Madan. “Together, our rich set of S3, HDFS, and POSIX APIs enables true storage-agnostic and multi-cloud deployments. The data migration capabilities further eliminate vendor lock-in and provide organizations with the flexibility to choose whatever they want to run their applications or store their data.”
Use Cases
Alluxio's primary use cases are to serve data-driven applications, whether large-scale analytics or ML/DL. Increasingly, optimizing resource spending across a hot and a cold tier of storage is gaining popularity as a use case. The ability to automate data movement with no awareness or impact on applications accessing the data being moved is the key here.
Customer Feedback
“At Uber, we run Alluxio to accelerate all sorts of business-critical analytics queries at a large scale,” said Chen Liang, Senior Software Engineer, Uber’s interactive analytics team. “Alluxio provides consistent performance in our big data processing use cases. As compute-storage separation continues to be the trend along with containerization in big data, we believe a unified layer that bridges the compute and storage like Alluxio will continue to play a key role.”
Features & Capabilities
Alluxio 2.8 Community and Enterprise Edition features new capabilities, including:
Enhanced S3 API with Metadata Tagging
Alluxio 2.8 enhances the support for the S3 RESTful API with metadata tagging capabilities. With S3 API, applications can communicate with Alluxio without a custom driver or any additional configuration. By using S3 API, data-driven applications, end-users, and admins can seamlessly and rapidly onboard Alluxio for new uses. Metadata operations can be achieved through the S3 object and bucket tagging APIs.
Data Encryption for Enterprise-grade Security
The Enterprise Edition of Alluxio 2.8 supports encryption of data in Alluxio managed storage at REST as an essential security feature. In conjunction with SSL, this new feature supports server-side encryption, ensuring data security. Alluxio now offers multiple encryption zones for data it manages to meet the demands of security. Data stored on the Alluxio worker is protected and decrypted on the server before being sent to a client.
Data Movement Across Heterogeneous Storage Systems
Alluxio 2.8 Enterprise Edition also improves the policy-based data management features, which facilitate data access and movement between heterogeneous storage systems for better performance and lower cost. Alluxio manages data placement between the different storage systems with predefined policies. With transparent storage options, organizations can choose whichever storage best suits their needs without undergoing complex manual data migrations.
Availability
Free downloads of Alluxio 2.8 open source Community Edition and trials of Alluxio Enterprise Edition are generally available here: https://www.alluxio.io/download/.