AWS Breaks Its Own Services with PostgreSQL 14 Upgrade
AWS recently ended standard support for PostgreSQL 13 on RDS, encouraging customers to upgrade to PostgreSQL 14 or later. However, this upgrade breaks AWS Glue, their managed ETL service, due to an incompatible password authentication scheme.
PostgreSQL 14 and AWS Glue Incompatibility
PostgreSQL 14 defaults to a more secure password authentication scheme, SCRAM-SHA-256, which AWS Glue cannot handle. This incompatibility has existed since PostgreSQL 14 shipped in 2021, but the deprecation of PostgreSQL 13 has removed the ability to avoid this problem.
Meanwhile, the Glue connection-testing infrastructure uses an internal driver that predates the newer authentication support, causing the “Test Connection” button to fail. Additionally, crawlers also fail, making it difficult for customers to verify their setup before trusting it with production data.
Technical Shape of the Catch-22
The incompatibility has been known since PostgreSQL 14 shipped, and the deprecation timeline for PG13 was announced in advance. However, the RDS and Glue teams apparently did not track each other’s developments, resulting in a gap between the services.
For example, customers can try downgrading password encryption on their database to the older, less secure standard, or bring their own JDBC driver, which disables connection testing and may not support all the features they want. Alternatively, they can rewrite their ETL workflows as Python shell jobs, but these solutions all have significant drawbacks.
Organizational Complexity and Customer Impact
The charitable read on this situation is that AWS has tens of thousands of engineers organized into hundreds of semi-autonomous service teams, making it difficult to track developments across teams. However, this does not excuse the lack of accountability and the significant impact on customers.
Customers who stayed on PG13 to avoid this problem are now facing Extended Support fees, which can be substantial. Furthermore, the response from AWS has been inadequate, with customers being offered paid alternatives that are not satisfactory.
Therefore, it is essential for customers to be aware of these potential issues and to plan accordingly. They should also demand more accountability from AWS and other cloud providers to ensure that their services are compatible and do not break each other.
Conclusion and Call to Action
In conclusion, the upgrade to PostgreSQL 14 has highlighted a significant issue with AWS Glue, and the response from AWS has been inadequate. Customers must be vigilant and demand more accountability from their cloud providers to avoid similar situations in the future.
Finally, we recommend that customers take a closer look at their cloud services and ensure that they are compatible with each other. They should also consider alternative solutions that can provide more flexibility and accountability.
