Introduction to Cloud SQL Troubleshooting
Google Cloud SQL is a fully-managed relational database service that supports MySQL, PostgreSQL, and SQL Server. While Google handles much of the operational overhead—such as backups, patching, and replication—developers and database administrators still encounter issues that require active troubleshooting. Understanding how to diagnose and resolve these common problems is essential for maintaining healthy, performant applications.
This tutorial walks through the most frequent Cloud SQL issues, their root causes, and practical solutions. Whether you are dealing with connection limits, slow queries, storage exhaustion, or replication lag, you will find actionable steps and code examples to get your database back on track.
Why Troubleshooting Cloud SQL Matters
Managed databases reduce operational burden, but they do not eliminate the need for vigilance. A misconfigured connection pool, an unoptimized query, or an unexpected spike in traffic can degrade performance or cause outages. Because Cloud SQL abstracts away the underlying infrastructure, some traditional debugging techniques—like SSH-ing into the database host—are unavailable. This means you must rely on Cloud Monitoring, Cloud Logging, and database-native tools to identify and resolve problems.
Effective troubleshooting minimizes downtime, controls costs, and ensures a smooth experience for your end users. It also helps you make informed decisions about scaling, instance sizing, and architecture changes.
Common Issue 1: Connection Exhaustion
One of the most frequent Cloud SQL problems is running out of available database connections. Each Cloud SQL instance has a maximum connection limit determined by its machine type and database engine. When applications open more connections than the instance can handle, new connection attempts fail with errors such as too many connections or connection refused.
Diagnosing Connection Issues
Start by checking the current connection count and the instance's configured limit. For PostgreSQL, you can run:
SELECT count(*) FROM pg_stat_activity;
SHOW max_connections;
For MySQL, use:
SHOW STATUS LIKE 'Threads_connected';
SHOW VARIABLES LIKE 'max_connections';
You can also monitor connections over time using Cloud Monitoring. The metric cloudsql.googleapis.com/database/mysql/connections (or the PostgreSQL equivalent) provides a historical view of connection usage.
Solutions for Connection Exhaustion
- Use connection pooling: Libraries like PgBouncer (PostgreSQL) or ProxySQL (MySQL) maintain a pool of reusable connections, reducing the overhead of opening and closing connections for each request.
- Use the Cloud SQL Auth Proxy: The Cloud SQL Auth Proxy manages connections efficiently and provides secure access without exposing your database's IP address.
- Close idle connections: Ensure your application closes connections when they are no longer needed. Many ORM frameworks support idle connection timeouts.
- Scale up the instance: If your workload genuinely requires more connections, upgrade to a larger machine type with a higher connection limit.
Here is an example of configuring a connection pool using Python's sqlalchemy with PostgreSQL:
from sqlalchemy import create_engine
from sqlalchemy.pool import QueuePool
engine = create_engine(
"postgresql+psycopg2://user:password@127.0.0.1:5432/mydb",
poolclass=QueuePool,
pool_size=10,
max_overflow=5,
pool_timeout=30,
pool_recycle=1800,
)
The pool_recycle parameter is particularly important for Cloud SQL, as it prevents the pool from holding stale connections that the server may have closed after a period of inactivity.
Common Issue 2: Slow Queries and Performance Degradation
Slow queries are another common source of frustration. They can cause high CPU utilization, increased latency, and even cascading failures if they block other operations. In Cloud SQL, slow queries often stem from missing indexes, inefficient query patterns, or insufficient instance resources.
Identifying Slow Queries
Enable the slow query log to capture queries that exceed a specified execution time. For MySQL, set the long_query_time flag:
gcloud sql instances patch my-instance \
--database-flags long_query_time=1,slow_query_log=ON
For PostgreSQL, enable log_min_duration_statement:
gcloud sql instances patch my-instance \
--database-flags log_min_duration_statement=1000
This logs any statement that takes longer than 1,000 milliseconds (1 second). You can view the logs in Cloud Logging by filtering for cloudsql.googleapis.com/postgres.log or the MySQL equivalent.
Analyzing Query Plans
Once you identify a slow query, use EXPLAIN ANALYZE to understand its execution plan. For PostgreSQL:
EXPLAIN ANALYZE
SELECT orders.id, customers.name
FROM orders
JOIN customers ON orders.customer_id = customers.id
WHERE orders.created_at > '2024-01-01'
ORDER BY orders.created_at DESC
LIMIT 100;
Look for sequential scans on large tables, nested loops with high row estimates, or sort operations that spill to disk. These are signs that an index is missing or the query needs rewriting.
Adding Indexes
If the execution plan reveals a sequential scan on a filtered column, adding an index can dramatically improve performance:
CREATE INDEX idx_orders_created_at
ON orders (created_at DESC);
CREATE INDEX idx_orders_customer_id
ON orders (customer_id);
For composite queries, consider a composite index that covers multiple columns in the order they are used:
CREATE INDEX idx_orders_customer_created
ON orders (customer_id, created_at DESC);
Best Practices for Query Performance
- Avoid SELECT *: Retrieve only the columns you need to reduce I/O and memory usage.
- Paginate large result sets: Use
LIMITandOFFSET, or better yet, keyset pagination for large datasets. - Monitor query statistics: Use
pg_stat_statements(PostgreSQL) or the Performance Schema (MySQL) to track query execution statistics over time. - Vacuum and analyze regularly: For PostgreSQL, ensure autovacuum is running properly to reclaim space and update planner statistics.
To enable pg_stat_statements on a Cloud SQL PostgreSQL instance:
gcloud sql instances patch my-instance \
--database-flags cloudsql.pg_stat_statements=on
Then query the view to find the most time-consuming queries:
SELECT query, calls, total_exec_time, mean_exec_time, rows
FROM pg_stat_statements
ORDER BY total_exec_time DESC
LIMIT 10;
Common Issue 3: Storage Exhaustion
Cloud SQL instances have a configured storage capacity. When storage reaches 95% utilization, the instance automatically switches to read-only mode to protect data integrity. In read-only mode, all write operations fail, which can break your application.
Detecting Storage Issues
Monitor the cloudsql.googleapis.com/database/disk/bytes_used metric in Cloud Monitoring. Set up an alerting policy to notify you when storage usage exceeds 80%:
gcloud alpha monitoring policies create \
--policy-from-file=storage-alert.json
Example storage-alert.json:
{
"displayName": "Cloud SQL Storage Alert",
"conditions": [
{
"displayName": "Disk usage above 80%",
"conditionThreshold": {
"filter": "resource.type=\"cloudsql_database\" AND metric.type=\"cloudsql.googleapis.com/database/disk/bytes_used\"",
"comparison": "COMPARISON_GT",
"thresholdValue": 0.8,
"duration": "300s",
"aggregations": [
{
"alignmentPeriod": "60s",
"perSeriesAligner": "ALIGN_MEAN"
}
]
}
}
],
"combiner": "OR",
"notificationChannels": [
"projects/my-project/notificationChannels/12345"
]
}
Resolving Storage Exhaustion
- Enable storage auto-increase: Cloud SQL can automatically increase storage capacity when it runs low. Enable this feature to prevent unexpected read-only mode.
- Manually increase storage: If auto-increase is disabled, you can manually increase the storage allocation.
- Clean up old data: Archive or delete old records, drop unused tables, and remove temporary files.
- Run VACUUM FULL (PostgreSQL): This reclaims space from dead tuples, but it locks the table and should be run during a maintenance window.
To enable storage auto-increase:
gcloud sql instances patch my-instance \
--storage-auto-increase
To manually increase storage:
gcloud sql instances patch my-instance \
--storage-size=200GB
Common Issue 4: High CPU and Memory Usage
When a Cloud SQL instance consistently runs at high CPU or memory utilization, query latency increases and the instance may become unresponsive. This issue often correlates with slow queries, but it can also result from insufficient instance sizing or runaway processes.
Investigating Resource Usage
Use Cloud Monitoring to examine CPU and memory metrics over time. Key metrics include:
cloudsql.googleapis.com/database/cpu/utilizationcloudsql.googleapis.com/database/memory/utilizationcloudsql.googleapis.com/database/memory/page_cache_utilization_ratio
For PostgreSQL, query pg_stat_activity to identify active queries consuming resources:
SELECT pid, now() - pg_stat_activity.query_start AS duration,
query, state
FROM pg_stat_activity
WHERE state = 'active'
ORDER BY duration DESC;
To terminate a long-running query that is consuming resources:
SELECT pg_terminate_backend(12345);
Replace 12345 with the actual PID from the query above.
Addressing High Resource Usage
- Optimize queries: As discussed in the slow queries section, indexes and query rewrites can reduce CPU load.
- Scale up the instance: Upgrade to a machine type with more vCPUs and memory if your workload has outgrown the current configuration.
- Use read replicas: Offload read-heavy workloads to read replicas to reduce the load on the primary instance.
- Tune database parameters: Adjust settings like
shared_buffers(PostgreSQL) orinnodb_buffer_pool_size(MySQL) to better utilize available memory.
To create a read replica:
gcloud sql instances create my-instance-replica \
--master-instance-name=my-instance \
--region=us-central1 \
--tier=db-custom-4-15360
Common Issue 5: Replication Lag
Read replicas in Cloud SQL asynchronously replicate data from the primary instance. Under normal conditions, replication lag is minimal—typically less than a second. However, heavy write workloads, large transactions, or network issues can cause lag to increase, leading to stale reads on replicas.
Monitoring Replication Lag
Monitor the cloudsql.googleapis.com/database/replication/replica_lag metric. Set up an alert for when lag exceeds an acceptable threshold, such as 30 seconds:
{
"displayName": "Replication Lag Alert",
"conditions": [
{
"displayName": "Replica lag above 30s",
"conditionThreshold": {
"filter": "resource.type=\"cloudsql_database\" AND metric.type=\"cloudsql.googleapis.com/database/replication/replica_lag\"",
"comparison": "COMPARISON_GT",
"thresholdValue": 30,
"duration": "300s",
"aggregations": [
{
"alignmentPeriod": "60s",
"perSeriesAligner": "ALIGN_MEAN"
}
]
}
}
],
"combiner": "OR"
}
For PostgreSQL, you can also check replication lag directly:
SELECT now() - pg_last_xact_replay_timestamp() AS replication_lag;
Reducing Replication Lag
- Batch large writes: Break large transactions into smaller batches to reduce the amount of WAL data that must be replicated at once.
- Upgrade the replica: Ensure the replica has sufficient resources to keep up with the primary. A replica with fewer resources than the primary may struggle under heavy load.
- Check for long-running queries on the replica: Long-running read queries on the replica can block replication replay. Identify and optimize them.
- Recreate the replica: If lag becomes persistent and unresolvable, drop and recreate the replica to start with a fresh snapshot.
To recreate a replica:
gcloud sql instances delete my-instance-replica
gcloud sql instances create my-instance-replica \
--master-instance-name=my-instance \
--region=us-central1 \
--tier=db-custom-4-15360
Common Issue 6: Connectivity and Authentication Errors
Connecting to Cloud SQL can fail for several reasons: incorrect network configuration, firewall rules, IAM permission issues, or expired SSL certificates. These errors typically manifest as connection timeouts or authentication failures.
Private IP Connectivity
If you are using a private IP connection, ensure your Cloud SQL instance is connected to a VPC network via a VPC peering connection. Verify the peering status:
gcloud compute networks peerings list \
--network=my-vpc-network
If the peering state is not ACTIVE, you may need to recreate the connection or check for conflicting routes.
Public IP and Authorized Networks
For public IP connections, add your client's IP address to the authorized networks list:
gcloud sql instances patch my-instance \
--authorized-networks=203.0.113.0/24
Using the Cloud SQL Auth Proxy
The Cloud SQL Auth Proxy is the recommended method for connecting to Cloud SQL from applications running outside Google Cloud. It handles authentication, encryption, and connection management automatically.
./cloud-sql-proxy my-project:us-central1:my-instance \
--credentials-file=key.json \
--port=5432
Then connect to 127.0.0.1:5432 as if it were your Cloud SQL instance.
SSL Certificate Issues
If you are using SSL connections and encounter certificate errors, the server certificate may have expired. You can rotate the certificate:
gcloud sql ssl-certs create new-cert cert.pem \
--instance=my-instance
Update your application to use the new certificate, then delete the old one:
gcloud sql ssl-certs delete old-cert-sha1 \
--instance=my-instance
Be cautious when rotating certificates, as deleting a certificate that is still in use will break existing connections. Always deploy the new certificate first.
Common Issue 7: Backup and Restoration Failures
Cloud SQL provides automated and on-demand backups. Occasionally, backups may fail due to instance state, storage issues, or configuration problems. Similarly, restoration operations can fail if the target instance configuration is incompatible.
Checking Backup Status
List recent backups and their status:
gcloud sql backups list --instance=my-instance
For detailed information about a specific backup:
gcloud sql backups describe BACKUP_ID \
--instance=my-instance
Creating an On-Demand Backup
gcloud sql backups create --instance=my-instance
Restoring from a Backup
To restore from a backup to the same instance:
gcloud sql instances restore-backup BACKUP_ID \
--restore-instance=my-instance
To perform a point-in-time recovery (PostgreSQL), create a new instance from a specific time:
gcloud sql instances clone my-instance my-instance-pitr \
--point-in-time=2024-06-15T10:00:00Z
Best Practices for Backups
- Enable automated backups: Ensure automated backups are enabled with a retention period that meets your recovery requirements.
- Test restores regularly: A backup is only as good as your ability to restore from it. Periodically test restoring to a separate instance.
- Enable point-in-time recovery: For production databases, enable PITR to recover from accidental data deletion or corruption.
- Monitor backup failures: Set up alerts for backup failure events in Cloud Logging.
To enable automated backups and point-in-time recovery:
gcloud sql instances patch my-instance \
--backup-start-time=02:00 \
--enable-point-in-time-recovery
Best Practices for Cloud SQL Operations
Beyond addressing specific issues, following operational best practices helps prevent problems before they occur. Here are key recommendations:
Monitoring and Alerting
Set up comprehensive monitoring for all critical metrics. At minimum, create alerts for:
- CPU utilization above 80%
- Memory utilization above 85%
- Disk usage above 80%
- Replication lag above 30 seconds
- Connection count above 80% of the maximum
- Backup failures
Database Maintenance
For PostgreSQL, ensure autovacuum is properly configured. Monitor dead tuple counts and vacuum frequency:
SELECT relname, n_dead_tup, n_live_tup,
round(n_dead_tup::float / n_live_tup::float * 100, 2) AS dead_ratio
FROM pg_stat_user_tables
WHERE n_live_tup > 0
ORDER BY dead_ratio DESC
LIMIT 10;
If autovacuum is not keeping up, consider tuning autovacuum parameters or running manual vacuums during off-peak hours:
VACUUM (ANALYZE, VERBOSE) large_table;
Security
- Use IAM database authentication: This eliminates the need to manage database passwords and integrates with Google Cloud IAM.
- Enforce SSL/TLS: Require SSL for all connections to encrypt data in transit.
- Use private IP: Keep your database off the public internet whenever possible.
- Regularly rotate credentials: Use Secret Manager to store and rotate database credentials automatically.
Performance Tuning
Regularly review and tune database flags. Common PostgreSQL flags to consider:
gcloud sql instances patch my-instance \
--database-flags \
shared_buffers=6GB,\
work_mem=64MB,\
maintenance_work_mem=512MB,\
effective_cache_size=18GB,\
random_page_cost=1.1
Adjust these values based on your instance size and workload characteristics. Always test changes in a non-production environment first.
Conclusion
Troubleshooting Cloud SQL effectively requires a combination of monitoring, database-native diagnostics, and an understanding of common failure modes. By proactively monitoring key metrics, optimizing queries, managing connections, and following operational best practices, you can prevent most issues before they impact your users. When problems do arise, the tools and techniques covered in this tutorial—Cloud Monitoring, slow query logs, execution plans, and the gcloud CLI—provide a systematic approach to diagnosis and resolution. Remember that managed services still require active management; investing time in understanding your database's behavior pays dividends in reliability, performance, and cost efficiency.