Databricks and AWS are strengthening their partnership to bolster GenAI capabilities. The data and AI specialist will use AWS Trainium processors to accelerate the development of LLMs within its public cloud environment.
The collaboration aims to enable customers to accelerate the development of custom LLMs within AWS using the data and AI specialist’s Mosaic AI platform hosted within the public cloud environment.
Specifically, this involves Databrick’s new Mosaic AI capabilities, including Mosaic AI Model Serving. This solution supports LLMs from several vendors, including the wide range available in AWS Bedrock.
Utilizing AWS Trainium chips
Users of the Mosaic AI development platform can deploy AWS Trainium chips as the processor of choice for training and running Databricks Mosaic AI LLMs on AWS.
The cloud provider’s AI chips can be utilized for pre-training, fine-tuning, enriching, and running aware models based on proprietary data. AWS’s infrastructure is expected to further support this with scalability, performance, and necessary security measures, all at a reduced cost.
Ultimately, according to Databricks and AWS, this collaboration should yield customized and production-ready GenAI applications.
Expanded collaboration
The further expanded partnership between the two entities also includes new Databricks integrations to be made available through the AWS Marketplace. In the future, the parties envision expanding the collaboration to include custom model optimization and enhanced security through the combination of Mosaic AI and AWS Trainium chips.
For migration functionality and modernizing GenAI applications, both companies will work closely with integrators to migrate data infrastructures from on-premises environments to AWS and modernize customers’ data platforms. The two sides will also introduce tailored solutions and building blocks for specific business sectors.
Other potential solutions and applications in the pipeline include improved usability, joint GenAI go-to-market programs, and increased joint marketing activities.
The Databricks Data Intelligence Platform is available on AWS and has received a FedRAMP High Agency Authority to Operate for cloud services on AWS GovCloud.
Also read: With Databricks Apps, business users get more out of data