I was able to connect to my local MySQL instance after just setting the server name to 127.0.0.1 instead of localhost and in the Advanced... settings set port to 3306 .
I originally tried to set my server name to localhost:3306 and 127.0.0.1 with no luck.
Additionally here is what my connection looks like in settings.json in Azure Data Studio:

"datasource.connections": [
    
        {
            "options": {
                "connectionName": "",
                "host": "127.0.0.1",
                "dbname": "",
                "user": "root",
                "password": "",
                "port": "3306",
                "connectTimeout": "10",
                "ssl": "require",
                "groupId": "C777F06B-202E-4480-B475-FA416154D458",
                "databaseDisplayName": ""
            },
            "groupId": "C777F06B-202E-4480-B475-FA416154D458",
            "providerName": "MySQL",
            "savePassword": true,
            "id": "a3b754b2-248a-4550-ba65-e099e8246d07"
        }
    ]

As I am writing, there is no native support for MySQL database connection. You can vote for the feature on Database Support MySQL Request . However, because Azure Data Studio supports python, you can connect to MySQL database using python libraries.

This feature is now available in preview (as of writing). You need to install the MySQL extension. Find the documentation and steps here Use Azure Data Studio to connect and query MySQL