Tech • AI • Robotics • Game
The New S3 Features Most Engineers Missed
7/10TL;DR
Amazon S3 has expanded beyond standard object buckets with S3 Tables for serverless Apache Iceberg data lakes and S3 Vector Buckets for embedding storage, reflecting AWS’s push to move analytics and AI workloads into managed S3-native formats.
KEY POINTS
A 20-year-old service adds new roles
Amazon S3, launched in 2006, remains one of AWS’s foundational services and is still widely used for storing files such as PDFs, images and CSV datasets. It also became a common base for enterprise data lakes, although misconfigured public bucket policies have repeatedly exposed sensitive data. In recent years, AWS has broadened S3 beyond general-purpose storage with more specialized bucket types.
Why traditional buckets were not enough
Standard S3 buckets worked well as raw storage, but data engineering teams often had to layer additional tools on top to organize data and query it through Hive-style tables. That approach created operational and governance challenges, especially as compliance and metadata management became more demanding. The rise of Apache Iceberg addressed many of those issues by offering an open table format better suited to modern lakehouse workloads.
S3 Tables target Iceberg-based lakehouses
AWS introduced S3 Tables in 2024 as a managed, serverless way to store data in Iceberg format. The service handles metadata, maintenance and compaction automatically, reducing the manual work usually required to run large analytical tables on object storage. Unlike a normal S3 bucket, users do not manage or browse the underlying files directly in the same way.
Built for analytics services like Athena
S3 Tables are designed to integrate with Amazon Athena, allowing teams to create and query Iceberg tables without maintaining the storage layout themselves. A typical setup can include separate namespaces for projects or business domains such as orders data. AWS also provides migration paths for organizations that already built data lakes on conventional S3 buckets and want to convert them to Iceberg-based structures.
Configuration is minimal
Creating a table bucket requires little more than a bucket name, a storage class choice such as Intelligent-Tiering, and optional encryption settings. After that, tables can be created from Athena, with data written into the managed Iceberg layout. The pitch is straightforward: teams keep the scale and durability of S3 while offloading table operations to AWS.
Vector Buckets bring AI storage into S3
AWS added S3 Vector Buckets in 2025 to store vector embeddings, a key building block for AI agents, retrieval systems and RAG applications. Embeddings are numerical representations used by machine learning systems to find related information, and they are typically stored in specialized vector databases. With vector buckets, AWS is positioning S3 as a serverless alternative for some of that infrastructure.
Indexes, dimensions and similarity search
Inside a vector bucket, users create one or more vector indexes and define settings such as embedding dimension and the distance metric, including cosine similarity. Multiple indexes can sit within a single bucket for different projects, experiments or subsets of data. That structure is meant to support use cases ranging from enterprise knowledge retrieval to product-specific AI search.
A shift from generic storage to managed purpose-built services
The broader message is that S3 is no longer just a place to dump files. For teams still storing Iceberg datasets or embeddings in ordinary buckets and maintaining their own layers on top, S3 Tables and S3 Vector Buckets offer AWS-managed alternatives. The tradeoff is less direct control over underlying files in exchange for lower operational overhead.
CONCLUSION
Amazon S3 is increasingly being positioned as a managed platform for both analytics and AI, not just basic object storage. The main decision for organizations is whether newer S3-native services can replace manual architectures built on traditional buckets.
Ask a question