15

How to Troubleshoot Your SDK Connection using SDK Doctor & Couchbase Alterna...

 3 years ago
source link: https://blog.couchbase.com/troubleshoot-sdk-connection-with-sdk-doctor-alternate-addresses/
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
neoserver,ios ssh client

How to Troubleshoot Your SDK Connection using SDK Doctor & Couchbase Alternate Addresses

It’s a common experience to run into network connection errors when you’re developing an application – particularly if you’re connecting from your local computer SDK to a remote cluster.

You’re likely to run into situations where your application can connect to the server but you get errors related to the network connection or IP addresses between the nodes or your app and the Couchbase SDK.

This article discusses how you can troubleshoot and fix these errors. The example we’ll walkthrough below creates a cluster in Azure to demonstrate the initial conditions, but it should be applicable to other installations as well.

We’ll go through each step in order to replicate the error. First, we’ll install Couchbase in Azure, then run SDK Doctor to see how the error manifests itself, then update the alternate addresses, and finally run the SDK Doctor again to see the successful response.

Okay, let’s dive in.

Install Your Couchbase Cluster in Azure

Here are your steps for installing a Couchbase cluster in Azure.

  1. Log into the Microsoft Azure Marketplace
  2. Search for Couchbase
  3. Choose Couchbase Server and Sync Gateway
  4. Click Get It Now
  5. Choose BYOL
  6. Click Continue
  7. Click Create
  8. Fill out form
  9. Admin: asdf88495 < Update this as appropriate
  10. Password: Password1!@#$ < Update this as appropriate
  11. Complete configuration forms as appropriate
  12. Click Create

The process runs and looks like so once you’ve completed all the steps:

Azure completed process after adding Couchbase

Update Firewall Rules

You likely need to update firewall rules to allow your local computer to connect to the remote Azure cluster. Here are the steps:

1. Go to Resource Group

2. Click on Virtual machine scale set

Microsoft Azure virtual machine scale set

3. Click on Networking

Couchbase on Azure Networking option

4. Click Add inbound port rule

Azure add inbound port rule button

5. Update the configuration to use your local network information. Warning: This is very open for the purposes of this demonstration, you should follow your security guidelines when configuring the firewall.

Local network firewall configuration

Confirm Couchbase Is Up and Running

Your next major step is to confirm that Couchbase is up and running. Here’s how you do that:

1. Go to instances to get an IP address and connect to the Couchbase control panel.

Couchbase control panel using IP address

2. Get the IP address for one of your servers.

Couchbase server IP address

3. In this example, the login is http://40.80.151.197:8091/ui/index.html.

4. Login with the info used to create the admin account from the “Install Your Couchbase Cluster in Azure” section above. This was where you created the cluster in Azure and completed the form.

5. In Couchbase, add sample data to use for this demonstration. Follow the instructions in the Couchbase documentation for adding data from sample buckets.

Introducing the SDK Doctor

The Couchbase SDK doctor is a tool to diagnose application-server-side connectivity issues with your Couchbase cluster. It makes the same connections to the Couchbase Server cluster that Couchbase SDKs make during bootstrapping, and then reports on the state of the connections made – giving diagnostic information that helps you solve puzzling network issues.

Here is the full set of the instructions for using the SDK Doctor with Couchbase Server.

1. Running the SDK Doctor to Spot Errors

Following the instructions above – where you collected the IP address for your Couchbase cluster – get the IP addresses for each of the other nodes in your cluster as well. You’ll use the IP addresses in the connection string for the SDK Doctor.

The command below uses the IP addresses for the nodes in your cluster in addition to the sample data bucket name. You will need to update this information based on your own cluster IPs and the specific sample bucket you installed.

Now, run the SDK Doctor.

You’ll probably get a bunch of error messages – as you can see below – especially if you are running SDK Doctor from a local computer and not on the same network as the nodes/cluster/VMs. You’ll see the ERROR and Failed to connect messages. Notice the IP addresses are the internal 10 addresses, not the 40 or 13.

You may need to double-click the code sample below to see the full output (depending on your device).

.\sdk-doctor-windows.exe diagnose couchbase://40.80.151.197,40.80.146.52,13.82.230.238/beer-sample  -u asdf88495 -p Password1!@#$
|====================================================================|
|          ___ ___  _  __   ___   ___   ___ _____ ___  ___           |
|         / __|   \| |/ /__|   \ / _ \ / __|_   _/ _ \| _ \          |
|         \__ \ |) | ' <___| |) | (_) | (__  | || (_) |   /          |
|         |___/___/|_|\_\  |___/ \___/ \___| |_| \___/|_|_\          |
|====================================================================|
Note: Diagnostics can only provide accurate results when your cluster
is in a stable state.  Active rebalancing and other cluster configuration
changes can cause the output of the doctor to be inconsistent or in the
worst cases, completely incorrect.
13:52:58.412 INFO ▶ Parsing connection string `couchbase://40.80.151.197,40.80.146.52,13.82.230.238/beer-sample`
13:52:58.413 INFO ▶ Connection string identifies the following CCCP endpoints:
13:52:58.413 INFO ▶   1. 40.80.151.197:11210
13:52:58.414 INFO ▶   2. 40.80.146.52:11210
13:52:58.414 INFO ▶   3. 13.82.230.238:11210
13:52:58.414 INFO ▶ Connection string identifies the following HTTP endpoints:
13:52:58.414 INFO ▶   1. 40.80.151.197:8091
13:52:58.414 INFO ▶   2. 40.80.146.52:8091
13:52:58.414 INFO ▶   3. 13.82.230.238:8091
13:52:58.415 INFO ▶ Connection string specifies bucket `beer-sample`
13:52:58.415 INFO ▶ Performing DNS lookup for host `40.80.151.197`
13:52:58.415 INFO ▶ Performing DNS lookup for host `40.80.146.52`
13:52:58.415 INFO ▶ Performing DNS lookup for host `13.82.230.238`
13:52:58.415 INFO ▶ Attempting to connect to cluster via CCCP
13:52:58.416 INFO ▶ Attempting to fetch config via cccp from `40.80.151.197:11210`
13:52:58.631 INFO ▶ Attempting to fetch config via cccp from `40.80.146.52:11210`
13:52:58.846 INFO ▶ Attempting to fetch config via cccp from `13.82.230.238:11210`
13:52:59.076 WARN ▶ Bootstrap host `40.80.151.197` is not using the canonical node hostname of `10.0.0.4`.  This is not necessarily an error, but has been known to result in strange and challenging to diagnose errors when DNS entries are reconfigured.
13:52:59.077 WARN ▶ Bootstrap host `40.80.146.52` is not using the canonical node hostname of `10.0.0.5`.  This is not neccessarily an error, but has been known to result in strange and challenging to diagnose errors when DNS entries are reconfigured.
13:52:59.079 WARN ▶ Bootstrap host `13.82.230.238` is not using the canonical node hostname of `10.0.0.6`.  This is not neccessarily an error, but has been known to result in strange and challenging to diagnose errors when DNS entries are reconfigured.
13:52:59.080 INFO ▶ Selected the following network type: default
13:52:59.080 INFO ▶ Identified the following nodes:
13:52:59.081 INFO ▶   [0] 10.0.0.4
13:52:59.081 INFO ▶                indexScan:  9101,      indexStreamInit:  9103,   indexStreamCatchup:  9104
13:52:59.081 INFO ▶                    kvSSL: 11207,            projector:  9999,                 mgmt:  8091
13:52:59.082 INFO ▶                  mgmtSSL: 18091,           indexAdmin:  9100,                 n1ql:  8093
13:52:59.082 INFO ▶                indexHttp:  9102,           indexHttps: 19102,                   kv: 11210
13:52:59.083 INFO ▶                     cbas:  8095,          eventingSSL: 18096,           ftsGRPCSSL: 19130
13:52:59.083 INFO ▶                   ftsSSL: 18094,              ftsGRPC:  9130,     indexStreamMaint:  9105
13:52:59.084 INFO ▶                     capi:  8092,              n1qlSSL: 18093,              cbasSSL: 18095
13:52:59.084 INFO ▶            eventingDebug:  9140,                  fts:  8094,    eventingAdminPort:  8096
13:52:59.085 INFO ▶                  capiSSL: 18092
13:52:59.085 INFO ▶   [1] 10.0.0.5
13:52:59.086 INFO ▶               ftsGRPCSSL: 19130,           indexHttps: 19102,                 cbas:  8095
13:52:59.086 INFO ▶        eventingAdminPort:  8096,        eventingDebug:  9140,           indexAdmin:  9100
13:52:59.087 INFO ▶          indexStreamInit:  9103,     indexStreamMaint:  9105,              mgmtSSL: 18091
13:52:59.087 INFO ▶                  cbasSSL: 18095,               ftsSSL: 18094,                 n1ql:  8093
13:52:59.088 INFO ▶                       kv: 11210,                 capi:  8092,              capiSSL: 18092
13:52:59.088 INFO ▶                  ftsGRPC:  9130,            indexScan:  9101,            indexHttp:  9102
13:52:59.089 INFO ▶       indexStreamCatchup:  9104,                kvSSL: 11207,                 mgmt:  8091
13:52:59.089 INFO ▶              eventingSSL: 18096,                  fts:  8094,            projector:  9999
13:52:59.089 INFO ▶                  n1qlSSL: 18093
13:52:59.090 INFO ▶   [2] 10.0.0.6
13:52:59.090 INFO ▶                  mgmtSSL: 18091,                 cbas:  8095,              cbasSSL: 18095
13:52:59.091 INFO ▶        eventingAdminPort:  8096,               ftsSSL: 18094,           indexAdmin:  9100
13:52:59.091 INFO ▶       indexStreamCatchup:  9104,                 capi:  8092,              ftsGRPC:  9130
13:52:59.092 INFO ▶                indexHttp:  9102,     indexStreamMaint:  9105,                kvSSL: 11207
13:52:59.092 INFO ▶                projector:  9999,        eventingDebug:  9140,                  fts:  8094
13:52:59.093 INFO ▶               ftsGRPCSSL: 19130,      indexStreamInit:  9103,                 mgmt:  8091
13:52:59.093 INFO ▶              eventingSSL: 18096,            indexScan:  9101,           indexHttps: 19102
13:52:59.093 INFO ▶                       kv: 11210,              capiSSL: 18092,                 n1ql:  8093
13:52:59.094 INFO ▶                  n1qlSSL: 18093
13:52:59.095 INFO ▶ Fetching config from `http://10.0.0.4:8091`
13:53:01.096 INFO ▶ Failed to retreive cluster information (error: Get "http://10.0.0.4:8091/pools/default": context deadline exceeded (Client.Timeout exceeded while awaiting headers))
13:53:03.097 ERRO ▶ Failed to connect to Key Value service at `10.0.0.4:11210` (error: dial tcp 10.0.0.4:11210: i/o timeout)
13:53:05.098 ERRO ▶ Failed to connect to Management service at `10.0.0.4:8091` (error: Get "http://10.0.0.4:8091/": context deadline exceeded (Client.Timeout exceeded while awaiting headers))
13:53:07.099 ERRO ▶ Failed to connect to Views service at `10.0.0.4:8092` (error: Get "http://10.0.0.4:8092/": context deadline exceeded (Client.Timeout exceeded while awaiting headers))
13:53:09.101 ERRO ▶ Failed to connect to Query service at `10.0.0.4:8093` (error: Get "http://10.0.0.4:8093/": context deadline exceeded (Client.Timeout exceeded while awaiting headers))
13:53:11.109 ERRO ▶ Failed to connect to Search service at `10.0.0.4:8094` (error: Get "http://10.0.0.4:8094/": context deadline exceeded (Client.Timeout exceeded while awaiting headers))
13:53:13.110 ERRO ▶ Failed to connect to Analytics service at `10.0.0.4:8095` (error: Get "http://10.0.0.4:8095/": context deadline exceeded (Client.Timeout exceeded while awaiting headers))
13:53:15.110 ERRO ▶ Failed to connect to Key Value service at `10.0.0.5:11210` (error: dial tcp 10.0.0.5:11210: i/o timeout)
13:53:17.119 ERRO ▶ Failed to connect to Management service at `10.0.0.5:8091` (error: Get "http://10.0.0.5:8091/": context deadline exceeded (Client.Timeout exceeded while awaiting headers))
13:53:19.119 ERRO ▶ Failed to connect to Views service at `10.0.0.5:8092` (error: Get "http://10.0.0.5:8092/": context deadline exceeded (Client.Timeout exceeded while awaiting headers))
13:53:21.120 ERRO ▶ Failed to connect to Query service at `10.0.0.5:8093` (error: Get "http://10.0.0.5:8093/": context deadline exceeded (Client.Timeout exceeded while awaiting headers))
13:53:23.120 ERRO ▶ Failed to connect to Search service at `10.0.0.5:8094` (error: Get "http://10.0.0.5:8094/": context deadline exceeded (Client.Timeout exceeded while awaiting headers))
13:53:25.121 ERRO ▶ Failed to connect to Analytics service at `10.0.0.5:8095` (error: Get "http://10.0.0.5:8095/": dial tcp 10.0.0.5:8095: i/o timeout (Client.Timeout exceeded while awaiting headers))
13:53:27.121 ERRO ▶ Failed to connect to Key Value service at `10.0.0.6:11210` (error: dial tcp 10.0.0.6:11210: i/o timeout)
13:53:29.122 ERRO ▶ Failed to connect to Management service at `10.0.0.6:8091` (error: Get "http://10.0.0.6:8091/": context deadline exceeded (Client.Timeout exceeded while awaiting headers))
13:53:31.123 ERRO ▶ Failed to connect to Views service at `10.0.0.6:8092` (error: Get "http://10.0.0.6:8092/": context deadline exceeded (Client.Timeout exceeded while awaiting headers))
13:53:33.124 ERRO ▶ Failed to connect to Query service at `10.0.0.6:8093` (error: Get "http://10.0.0.6:8093/": context deadline exceeded (Client.Timeout exceeded while awaiting headers))
13:53:35.124 ERRO ▶ Failed to connect to Search service at `10.0.0.6:8094` (error: Get "http://10.0.0.6:8094/": context deadline exceeded (Client.Timeout exceeded while awaiting headers))
13:53:37.125 ERRO ▶ Failed to connect to Analytics service at `10.0.0.6:8095` (error: Get "http://10.0.0.6:8095/": context deadline exceeded (Client.Timeout exceeded while awaiting headers))
13:53:39.126 WARN ▶ Failed to perform KV connection performance analysis on `10.0.0.4:11210` (error: %!d(string=dial tcp 10.0.0.4:11210: i/o timeout))
13:53:41.127 WARN ▶ Failed to perform KV connection performance analysis on `10.0.0.5:11210` (error: %!d(string=dial tcp 10.0.0.5:11210: i/o timeout))
13:53:43.128 WARN ▶ Failed to perform KV connection performance analysis on `10.0.0.6:11210` (error: %!d(string=dial tcp 10.0.0.6:11210: i/o timeout))
13:53:43.128 INFO ▶ Diagnostics completed
Summary:
[WARN] Bootstrap host `40.80.151.197` is not using the canonical node hostname of `10.0.0.4`.  This is not neccessarily an error, but has been known to result in strange and challenging to diagnose errors when DNS entries are reconfigured.
[WARN] Bootstrap host `40.80.146.52` is not using the canonical node hostname of `10.0.0.5`.  This is not neccessarily an error, but has been known to result in strange and challenging to diagnose errors when DNS entries are reconfigured.
[WARN] Bootstrap host `13.82.230.238` is not using the canonical node hostname of `10.0.0.6`.  This is not neccessarily an error, but has been known to result in strange and challenging to diagnose errors when DNS entries are reconfigured.
[WARN] Failed to perform KV connection performance analysis on `10.0.0.4:11210` (error: %!d(string=dial tcp 10.0.0.4:11210: i/o timeout))
[WARN] Failed to perform KV connection performance analysis on `10.0.0.5:11210` (error: %!d(string=dial tcp 10.0.0.5:11210: i/o timeout))
[WARN] Failed to perform KV connection performance analysis on `10.0.0.6:11210` (error: %!d(string=dial tcp 10.0.0.6:11210: i/o timeout))
[ERRO] Failed to connect to Key Value service at `10.0.0.4:11210` (error: dial tcp 10.0.0.4:11210: i/o timeout)
[ERRO] Failed to connect to Management service at `10.0.0.4:8091` (error: Get "http://10.0.0.4:8091/": context deadline exceeded (Client.Timeout exceeded while awaiting headers))
[ERRO] Failed to connect to Views service at `10.0.0.4:8092` (error: Get "http://10.0.0.4:8092/": context deadline exceeded (Client.Timeout exceeded while awaiting headers))
[ERRO] Failed to connect to Query service at `10.0.0.4:8093` (error: Get "http://10.0.0.4:8093/": context deadline exceeded (Client.Timeout exceeded while awaiting headers))
[ERRO] Failed to connect to Search service at `10.0.0.4:8094` (error: Get "http://10.0.0.4:8094/": context deadline exceeded (Client.Timeout exceeded while awaiting headers))
[ERRO] Failed to connect to Analytics service at `10.0.0.4:8095` (error: Get "http://10.0.0.4:8095/": context deadline exceeded (Client.Timeout exceeded while awaiting headers))
[ERRO] Failed to connect to Key Value service at `10.0.0.5:11210` (error: dial tcp 10.0.0.5:11210: i/o timeout)
[ERRO] Failed to connect to Management service at `10.0.0.5:8091` (error: Get "http://10.0.0.5:8091/": context deadline exceeded (Client.Timeout exceeded while awaiting headers))
[ERRO] Failed to connect to Views service at `10.0.0.5:8092` (error: Get "http://10.0.0.5:8092/": context deadline exceeded (Client.Timeout exceeded while awaiting headers))
[ERRO] Failed to connect to Query service at `10.0.0.5:8093` (error: Get "http://10.0.0.5:8093/": context deadline exceeded (Client.Timeout exceeded while awaiting headers))
[ERRO] Failed to connect to Search service at `10.0.0.5:8094` (error: Get "http://10.0.0.5:8094/": context deadline exceeded (Client.Timeout exceeded while awaiting headers))
[ERRO] Failed to connect to Analytics service at `10.0.0.5:8095` (error: Get "http://10.0.0.5:8095/": dial tcp 10.0.0.5:8095: i/o timeout (Client.Timeout exceeded while awaiting headers))
[ERRO] Failed to connect to Key Value service at `10.0.0.6:11210` (error: dial tcp 10.0.0.6:11210: i/o timeout)
[ERRO] Failed to connect to Management service at `10.0.0.6:8091` (error: Get "http://10.0.0.6:8091/": context deadline exceeded (Client.Timeout exceeded while awaiting headers))
[ERRO] Failed to connect to Views service at `10.0.0.6:8092` (error: Get "http://10.0.0.6:8092/": context deadline exceeded (Client.Timeout exceeded while awaiting headers))
[ERRO] Failed to connect to Query service at `10.0.0.6:8093` (error: Get "http://10.0.0.6:8093/": context deadline exceeded (Client.Timeout exceeded while awaiting headers))
[ERRO] Failed to connect to Search service at `10.0.0.6:8094` (error: Get "http://10.0.0.6:8094/": context deadline exceeded (Client.Timeout exceeded while awaiting headers))
[ERRO] Failed to connect to Analytics service at `10.0.0.6:8095` (error: Get "http://10.0.0.6:8095/": context deadline exceeded (Client.Timeout exceeded while awaiting headers))
Found multiple issues, see listing above.

Using the Couchbase Alternate Address

Couchbase Server allows an alternate address to be assigned to any individual cluster-node, and an alternate port number to be assigned to any service running on that node.

The assigning of alternate addresses and port numbers – which can be accomplished by means of the CLI and the REST API – may facilitate communication with external applications that are not permitted to contact a cluster’s nodes directly. However, they may have access to a router or other networked entity that provides externally visible addresses on the nodes’ behalf.

Dive into the Couchbase documentation on using alternate addresses to learn more.

Add the Couchbase Alternate Address

There is a REST call to update the alternate address for your Couchbase cluster. (For more information, check out the Couchbase documentation on managing alternate addresses with the REST API.)

Next, you’ll want to update the alternate address for each IP address in the cluster. This maps the internal to the IP address that you specify.

curl -v -X PUT -u asdf88495:Password1!@#$ http://40.80.151.197:8091/node/controller/setupAlternateAddresses/external -d hostname=40.80.151.197

You should get a 200 response, as you can see below.

*   Trying 13.82.230.238...
* TCP_NODELAY set
* Connected to 13.82.230.238 (13.82.230.238) port 8091 (#0)
* Server auth using Basic with user 'asdf88495'
> PUT /node/controller/setupAlternateAddresses/external HTTP/1.1
> Host: 13.82.230.238:8091
> Authorization: Basic YXNkZjg4NDk1OlBhc3N3b3JkMSFAIyQ=
> User-Agent: curl/7.55.1
> Accept: */*
> Content-Length: 22
> Content-Type: application/x-www-form-urlencoded
* upload completely sent off: 22 out of 22 bytes
< HTTP/1.1 200 OK
< X-XSS-Protection: 1; mode=block
< X-Permitted-Cross-Domain-Policies: none
< X-Frame-Options: DENY
< X-Content-Type-Options: nosniff
< Server: Couchbase Server
< Pragma: no-cache
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
< Date: Mon, 28 Jun 2021 18:59:31 GMT
< Content-Length: 0
< Cache-Control: no-cache,no-store,must-revalidate

Repeat the command for the other hosts as well, like so:

curl -v -X PUT -u asdf88495:Password1!@#$ http://40.80.146.52:8091/node/controller/setupAlternateAddresses/external -d hostname=40.80.146.52
curl -v -X PUT -u asdf88495:Password1!@#$ http://13.82.230.238:8091/node/controller/setupAlternateAddresses/external -d hostname=13.82.230.238

2. Running the SDK Doctor to Confirm Success

Now, run the SDK Doctor again to confirm success. You should get something like this below:

You may need to double-click the code sample below to see the full output (depending on your device).

C:\david\bin\Couchbase> .\sdk-doctor-windows.exe diagnose couchbase://40.80.151.197,40.80.146.52,13.82.230.238/beer-sample  -u asdf88495 -p Password1!@#$
|====================================================================|
|          ___ ___  _  __   ___   ___   ___ _____ ___  ___           |
|         / __|   \| |/ /__|   \ / _ \ / __|_   _/ _ \| _ \          |
|         \__ \ |) | ' <___| |) | (_) | (__  | || (_) |   /          |
|         |___/___/|_|\_\  |___/ \___/ \___| |_| \___/|_|_\          |
|====================================================================|
Note: Diagnostics can only provide accurate results when your cluster
is in a stable state.  Active rebalancing and other cluster configuration
changes can cause the output of the doctor to be inconsistent or in the
worst cases, completely incorrect.
13:59:33.729 INFO ▶ Parsing connection string `couchbase://40.80.151.197,40.80.146.52,13.82.230.238/beer-sample`
13:59:33.730 INFO ▶ Connection string identifies the following CCCP endpoints:
13:59:33.730 INFO ▶   1. 40.80.151.197:11210
13:59:33.730 INFO ▶   2. 40.80.146.52:11210
13:59:33.730 INFO ▶   3. 13.82.230.238:11210
13:59:33.730 INFO ▶ Connection string identifies the following HTTP endpoints:
13:59:33.731 INFO ▶   1. 40.80.151.197:8091
13:59:33.731 INFO ▶   2. 40.80.146.52:8091
13:59:33.731 INFO ▶   3. 13.82.230.238:8091
13:59:33.731 INFO ▶ Connection string specifies bucket `beer-sample`
13:59:33.731 INFO ▶ Performing DNS lookup for host `40.80.151.197`
13:59:33.731 INFO ▶ Performing DNS lookup for host `40.80.146.52`
13:59:33.731 INFO ▶ Performing DNS lookup for host `13.82.230.238`
13:59:33.731 INFO ▶ Attempting to connect to cluster via CCCP
13:59:33.732 INFO ▶ Attempting to fetch config via cccp from `40.80.151.197:11210`
13:59:33.947 INFO ▶ Attempting to fetch config via cccp from `40.80.146.52:11210`
13:59:34.171 INFO ▶ Attempting to fetch config via cccp from `13.82.230.238:11210`
13:59:34.391 WARN ▶ Bootstrap host `40.80.151.197` is not using the canonical node hostname of `10.0.0.4`.  This is not neccessarily an error, but has been known to result in strange and challenging to diagnose errors when DNS entries are reconfigured.
13:59:34.392 WARN ▶ Bootstrap host `40.80.146.52` is not using the canonical node hostname of `10.0.0.5`.  This is not neccessarily an error, but has been known to result in strange and challenging to diagnose errors when DNS entries are reconfigured.
13:59:34.393 WARN ▶ Bootstrap host `13.82.230.238` is not using the canonical node hostname of `10.0.0.6`.  This is not neccessarily an error, but has been known to result in strange and challenging to diagnose errors when DNS entries are reconfigured.
13:59:34.394 INFO ▶ Selected the following network type: external
13:59:34.394 INFO ▶ Identified the following nodes:
13:59:34.394 INFO ▶   [0] 40.80.151.197
13:59:34.395 INFO ▶                projector:  9999,                 n1ql:  8093,              n1qlSSL: 18093
13:59:34.395 INFO ▶                  cbasSSL: 18095,           indexAdmin:  9100,                kvSSL: 11207
13:59:34.396 INFO ▶                     capi:  8092,              capiSSL: 18092,   indexStreamCatchup:  9104
13:59:34.396 INFO ▶                  mgmtSSL: 18091,    eventingAdminPort:  8096,          eventingSSL: 18096
13:59:34.397 INFO ▶                      fts:  8094,              ftsGRPC:  9130,        eventingDebug:  9140
13:59:34.397 INFO ▶                indexHttp:  9102,     indexStreamMaint:  9105,      indexStreamInit:  9103
13:59:34.398 INFO ▶               indexHttps: 19102,                   kv: 11210,                 mgmt:  8091
13:59:34.398 INFO ▶                     cbas:  8095,               ftsSSL: 18094,           ftsGRPCSSL: 19130
13:59:34.399 INFO ▶                indexScan:  9101
13:59:34.399 INFO ▶   [1] 40.80.146.52
13:59:34.400 INFO ▶                  capiSSL: 18092,              mgmtSSL: 18091,      indexStreamInit:  9103
13:59:34.400 INFO ▶       indexStreamCatchup:  9104,     indexStreamMaint:  9105,           indexHttps: 19102
13:59:34.400 INFO ▶                     n1ql:  8093,              n1qlSSL: 18093,              cbasSSL: 18095
13:59:34.401 INFO ▶            eventingDebug:  9140,          eventingSSL: 18096,                  fts:  8094
13:59:34.401 INFO ▶                  ftsGRPC:  9130,                 mgmt:  8091,                 cbas:  8095
13:59:34.402 INFO ▶        eventingAdminPort:  8096,            indexScan:  9101,                kvSSL: 11207
13:59:34.402 INFO ▶                     capi:  8092,            projector:  9999,               ftsSSL: 18094
13:59:34.403 INFO ▶               ftsGRPCSSL: 19130,           indexAdmin:  9100,            indexHttp:  9102
13:59:34.403 INFO ▶                       kv: 11210
13:59:34.403 INFO ▶   [2] 13.82.230.238
13:59:34.404 INFO ▶                       kv: 11210,            projector:  9999,          eventingSSL: 18096
13:59:34.404 INFO ▶          indexStreamInit:  9103,           indexHttps: 19102,     indexStreamMaint:  9105
13:59:34.405 INFO ▶                    kvSSL: 11207,              capiSSL: 18092,    eventingAdminPort:  8096
13:59:34.406 INFO ▶                  ftsGRPC:  9130,            indexScan:  9101,           ftsGRPCSSL: 19130
13:59:34.406 INFO ▶       indexStreamCatchup:  9104,                 n1ql:  8093,              n1qlSSL: 18093
13:59:34.407 INFO ▶                     mgmt:  8091,              mgmtSSL: 18091,               ftsSSL: 18094
13:59:34.407 INFO ▶                      fts:  8094,           indexAdmin:  9100,            indexHttp:  9102
13:59:34.407 INFO ▶                     capi:  8092,                 cbas:  8095,              cbasSSL: 18095
13:59:34.408 INFO ▶            eventingDebug:  9140
13:59:34.408 INFO ▶ Fetching config from `http://40.80.151.197:8091`
13:59:34.511 INFO ▶ Received cluster configuration, nodes list:
    "addressFamily": "inet",
    "alternateAddresses": {
      "external": {
        "hostname": "40.80.151.197",
        "ports": {
          "capi": 8092,
          "capiSSL": 18092,
          "kv": 11210,
          "mgmt": 8091,
          "mgmtSSL": 18091
    "clusterCompatibility": 393222,
    "clusterMembership": "active",
    "configuredHostname": "10.0.0.4:8091",
    "couchApiBase": "http://10.0.0.4:8092/",
    "couchApiBaseHTTPS": "https://10.0.0.4:18092/",
    "cpuCount": 4,
    "externalListeners": [
        "afamily": "inet",
        "nodeEncryption": false
        "afamily": "inet6",
        "nodeEncryption": false
    "hostname": "10.0.0.4:8091",
    "interestingStats": {
      "cmd_get": 0,
      "couch_docs_actual_disk_size": 17689291,
      "couch_docs_data_size": 11756324,
      "couch_spatial_data_size": 0,
      "couch_spatial_disk_size": 0,
      "couch_views_actual_disk_size": 22774,
      "couch_views_data_size": 11416,
      "curr_items": 2475,
      "curr_items_tot": 4897,
      "ep_bg_fetched": 0,
      "get_hits": 0,
      "mem_used": 22484624,
      "ops": 0,
      "vb_active_num_non_resident": 0,
      "vb_replica_curr_items": 2422
    "mcdMemoryAllocated": 22468,
    "mcdMemoryReserved": 22468,
    "memoryFree": 27792080896,
    "memoryTotal": 29450268672,
    "nodeEncryption": false,
    "nodeUUID": "616c0853d0e7cbacab9cbdd6ec7a51f7",
    "os": "x86_64-unknown-linux-gnu",
    "otpNode": "[email protected]",
    "ports": {
      "direct": 11210,
      "distTCP": 21100,
      "distTLS": 21150,
      "httpsCAPI": 18092,
      "httpsMgmt": 18091
    "recoveryType": "none",
    "services": [
      "cbas",
      "eventing",
      "fts",
      "index",
      "kv",
      "n1ql"
    "status": "healthy",
    "systemStats": {
      "allocstall": 0,
      "cpu_cores_available": 4,
      "cpu_stolen_rate": 0,
      "cpu_utilization_rate": 3.282828282828283,
      "mem_free": 27792080896,
      "mem_limit": 29450268672,
      "mem_total": 29450268672,
      "swap_total": 0,
      "swap_used": 0
    "thisNode": true,
    "uptime": "1122",
    "version": "6.6.2-9588-enterprise"
    "addressFamily": "inet",
    "alternateAddresses": {
      "external": {
        "hostname": "40.80.146.52",
        "ports": {
          "capi": 8092,
          "capiSSL": 18092,
          "kv": 11210,
          "mgmt": 8091,
          "mgmtSSL": 18091
    "clusterCompatibility": 393222,
    "clusterMembership": "active",
    "configuredHostname": "10.0.0.5:8091",
    "couchApiBase": "http://10.0.0.5:8092/",
    "couchApiBaseHTTPS": "https://10.0.0.5:18092/",
    "cpuCount": 4,
    "externalListeners": [
        "afamily": "inet",
        "nodeEncryption": false
        "afamily": "inet6",
        "nodeEncryption": false
    "hostname": "10.0.0.5:8091",
    "interestingStats": {
      "cmd_get": 0,
      "couch_docs_actual_disk_size": 17275589,
      "couch_docs_data_size": 11350638,
      "couch_spatial_data_size": 0,
      "couch_spatial_disk_size": 0,
      "couch_views_actual_disk_size": 22737,
      "couch_views_data_size": 11408,
      "curr_items": 2415,
      "curr_items_tot": 4862,
      "ep_bg_fetched": 0,
      "get_hits": 0,
      "mem_used": 22640944,
      "ops": 0,
      "vb_active_num_non_resident": 0,
      "vb_replica_curr_items": 2447
    "mcdMemoryAllocated": 22468,
    "mcdMemoryReserved": 22468,
    "memoryFree": 28102778880,
    "memoryTotal": 29450268672,
    "nodeEncryption": false,
    "nodeUUID": "94565e32208f2d93915124d49842258c",
    "os": "x86_64-unknown-linux-gnu",
    "otpNode": "[email protected]",
    "ports": {
      "direct": 11210,
      "distTCP": 21100,
      "distTLS": 21150,
      "httpsCAPI": 18092,
      "httpsMgmt": 18091
    "recoveryType": "none",
    "services": [
      "cbas",
      "eventing",
      "fts",
      "index",
      "kv",
      "n1ql"
    "status": "healthy",
    "systemStats": {
      "allocstall": 0,
      "cpu_cores_available": 4,
      "cpu_stolen_rate": 0,
      "cpu_utilization_rate": 2.763819095477387,
      "mem_free": 28102778880,
      "mem_limit": 29450268672,
      "mem_total": 29450268672,
      "swap_total": 0,
      "swap_used": 0
    "uptime": "1367",
    "version": "6.6.2-9588-enterprise"
    "addressFamily": "inet",
    "alternateAddresses": {
      "external": {
        "hostname": "13.82.230.238",
        "ports": {
          "capi": 8092,
          "capiSSL": 18092,
          "kv": 11210,
          "mgmt": 8091,
          "mgmtSSL": 18091
    "clusterCompatibility": 393222,
    "clusterMembership": "active",
    "configuredHostname": "10.0.0.6:8091",
    "couchApiBase": "http://10.0.0.6:8092/",
    "couchApiBaseHTTPS": "https://10.0.0.6:18092/",
    "cpuCount": 4,
    "externalListeners": [
        "afamily": "inet",
        "nodeEncryption": false
        "afamily": "inet6",
        "nodeEncryption": false
    "hostname": "10.0.0.6:8091",
    "interestingStats": {
      "cmd_get": 0,
      "couch_docs_actual_disk_size": 17363691,
      "couch_docs_data_size": 11432558,
      "couch_spatial_data_size": 0,
      "couch_spatial_disk_size": 0,
      "couch_views_actual_disk_size": 22756,
      "couch_views_data_size": 11415,
      "curr_items": 2413,
      "curr_items_tot": 4847,
      "ep_bg_fetched": 0,
      "get_hits": 0,
      "mem_used": 22550192,
      "ops": 0,
      "vb_active_num_non_resident": 0,
      "vb_replica_curr_items": 2434
    "mcdMemoryAllocated": 22468,
    "mcdMemoryReserved": 22468,
    "memoryFree": 28038606848,
    "memoryTotal": 29450268672,
    "nodeEncryption": false,
    "nodeUUID": "d66d1a929cde564fa27cec0d3ef56520",
    "os": "x86_64-unknown-linux-gnu",
    "otpNode": "[email protected]",
    "ports": {
      "direct": 11210,
      "distTCP": 21100,
      "distTLS": 21150,
      "httpsCAPI": 18092,
      "httpsMgmt": 18091
    "recoveryType": "none",
    "services": [
      "cbas",
      "eventing",
      "fts",
      "index",
      "kv",
      "n1ql"
    "status": "healthy",
    "systemStats": {
      "allocstall": 0,
      "cpu_cores_available": 4,
      "cpu_stolen_rate": 0,
      "cpu_utilization_rate": 1.754385964912281,
      "mem_free": 28038606848,
      "mem_limit": 29450268672,
      "mem_total": 29450268672,
      "swap_total": 0,
      "swap_used": 0
    "uptime": "1136",
    "version": "6.6.2-9588-enterprise"
13:59:34.706 INFO ▶ Successfully connected to Key Value service at `40.80.151.197:11210`
13:59:34.845 INFO ▶ Successfully connected to Management service at `40.80.151.197:8091`
13:59:34.933 INFO ▶ Successfully connected to Views service at `40.80.151.197:8092`
13:59:35.020 INFO ▶ Successfully connected to Query service at `40.80.151.197:8093`
13:59:35.108 INFO ▶ Successfully connected to Search service at `40.80.151.197:8094`
13:59:35.200 INFO ▶ Successfully connected to Analytics service at `40.80.151.197:8095`
13:59:35.377 INFO ▶ Successfully connected to Key Value service at `40.80.146.52:11210`
13:59:35.515 INFO ▶ Successfully connected to Management service at `40.80.146.52:8091`
13:59:35.619 INFO ▶ Successfully connected to Views service at `40.80.146.52:8092`
13:59:35.713 INFO ▶ Successfully connected to Query service at `40.80.146.52:8093`
13:59:35.803 INFO ▶ Successfully connected to Search service at `40.80.146.52:8094`
13:59:35.903 INFO ▶ Successfully connected to Analytics service at `40.80.146.52:8095`
13:59:36.074 INFO ▶ Successfully connected to Key Value service at `13.82.230.238:11210`
13:59:36.206 INFO ▶ Successfully connected to Management service at `13.82.230.238:8091`
13:59:36.308 INFO ▶ Successfully connected to Views service at `13.82.230.238:8092`
13:59:36.397 INFO ▶ Successfully connected to Query service at `13.82.230.238:8093`
13:59:36.487 INFO ▶ Successfully connected to Search service at `13.82.230.238:8094`
13:59:36.591 INFO ▶ Successfully connected to Analytics service at `13.82.230.238:8095`
13:59:37.207 INFO ▶ Memd Nop Pinged `40.80.151.197:11210` 10 times, 0 errors, 42ms min, 45ms max, 43ms mean
13:59:37.208 WARN ▶ Memcached service on `40.80.151.197:11210` on average took longer than 10ms (was: 43ms) to reply.  This is usually due to network-related issues, and could significantly affect application performance.
13:59:37.209 WARN ▶ Memcached service on `40.80.151.197:11210` maximally took longer than 20ms (was: 45ms) to reply. This is usually due to network-related issues, and could significantly affect application performance.
13:59:37.803 INFO ▶ Memd Nop Pinged `40.80.146.52:11210` 10 times, 0 errors, 42ms min, 42ms max, 42ms mean
13:59:37.804 WARN ▶ Memcached service on `40.80.146.52:11210` on average took longer than 10ms (was: 42ms) to reply.  This is usually due to network-related issues, and could significantly affect application performance.
13:59:37.804 WARN ▶ Memcached service on `40.80.146.52:11210` maximally took longer than 20ms (was: 42ms) to reply. This is usually due to network-related issues, and could significantly affect application performance.
13:59:38.423 INFO ▶ Memd Nop Pinged `13.82.230.238:11210` 10 times, 0 errors, 43ms min, 44ms max, 44ms mean
13:59:38.423 WARN ▶ Memcached service on `13.82.230.238:11210` on average took longer than 10ms (was: 44ms) to reply.  This is usually due to network-related issues, and could significantly affect application performance.
13:59:38.423 WARN ▶ Memcached service on `13.82.230.238:11210` maximally took longer than 20ms (was: 44ms) to reply. This is usually due to network-related issues, and could significantly affect application performance.
13:59:38.424 INFO ▶ Diagnostics completed
Summary:
[WARN] Bootstrap host `40.80.151.197` is not using the canonical node hostname of `10.0.0.4`.  This is not neccessarily an error, but has been known to result in strange and challenging to diagnose errors when DNS entries are reconfigured.
[WARN] Bootstrap host `40.80.146.52` is not using the canonical node hostname of `10.0.0.5`.  This is not neccessarily an error, but has been known to result in strange and challenging to diagnose errors when DNS entries are reconfigured.
[WARN] Bootstrap host `13.82.230.238` is not using the canonical node hostname of `10.0.0.6`.  This is not neccessarily an error, but has been known to result in strange and challenging to diagnose errors when DNS entries are reconfigured.
[WARN] Memcached service on `40.80.151.197:11210` on average took longer than 10ms (was: 43ms) to reply.  This is usually due to network-related issues, and could significantly affect application performance.
[WARN] Memcached service on `40.80.151.197:11210` maximally took longer than 20ms (was: 45ms) to reply. This is usually due to network-related issues, and could significantly affect application performance.
[WARN] Memcached service on `40.80.146.52:11210` on average took longer than 10ms (was: 42ms) to reply.  This is usually due to network-related issues, and could significantly affect application performance.
[WARN] Memcached service on `40.80.146.52:11210` maximally took longer than 20ms (was: 42ms) to reply. This is usually due to network-related issues, and could significantly affect application performance.
[WARN] Memcached service on `13.82.230.238:11210` on average took longer than 10ms (was: 44ms) to reply.  This is usually due to network-related issues, and could significantly affect application performance.
[WARN] Memcached service on `13.82.230.238:11210` maximally took longer than 20ms (was: 44ms) to reply. This is usually due to network-related issues, and could significantly affect application performance.
Found multiple issues, see listing above.

Success!

Conclusion

Connecting to Couchbase clusters in the cloud offers a lot of advantages. But sometimes – due to various network and security requirements – the remote and distributed nature of systems can cause complications between your app SDK and the Couchbase cluster.

By using the Couchbase SDK Doctor and Alternate Addresses, you can easily troubleshoot any complications that arise and fix connection problems right away.

Ready to try out Couchbase Server for yourself?

Take Couchbase 7 for a spin

About Joyk


Aggregate valuable and interesting links.
Joyk means Joy of geeK