Simple Things You Can Learn From Cassandra Nodetool (Monitor/Manage) For DC/OS
source link: https://blog.knoldus.com/learn-cassandra-nodetool-monitormanage/
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
As we know, DC/OS gave us minimal packages of services for running on apache mesos like cassandra. Cassandra services only give us the functionality for storing data , not other things like native tools as `nodetool`, `cqlsh`, `cassandra-stress` and more. Today, we are looking into the way for using `nodetool`, for monitoring DC/OS cassandra services nodes or cluster.
Brief:
Nodetool is a native command line interface for managing and monitoring cassandra cluster. While our cassandra services are running on DC/OS, we can manage and monitor cluster using nodetool. We need to execute nodetool on DC/OS using cassandra docker image on any of the DC/OS cassandra nodes by performing below steps:
- First, need to ssh any of DC/OS cassandra node: ssh @
- Second, execute the docker command for download cassandra docker image and pass nodetool command for execution as below:
$ sudo docker run -t --net=host cassandra:3.0 nodetool tablestats -H .
Note: Sometime nodetool commands may not work if Cassandra version and Nodetool version is incompatible e.g: Cassandra Version 3.0 and Nodetool version 3.10, nodetool will not be able to run all commands.
Below, we are looking into some of the important nodetool commands for monitoring perspective of cassandra cluster. But still, nodetool have lots of commands, which we are not discussing now. For reference please click on this link.
1. CASSANDRA CLUSTER STATUS
For looking into cassandra cluster health or status using node tool, we need to execute below command:
sudo docker run -t --net=host cassandra:3.0 nodetool -p 7199 status
OUTPUT:
— Address Load Tokens Owns (effective) Host ID Rack UN
23.26 GB 256 21.7%
rac1 UN
20.48 GB 256 19.7%
rac1 UN
21.62 GB 256 19.7%
rac1
- U indicate whether the node is UP.
- N indicates whether the node is NORMAL.
- DC/OS cassandra node IP address.
- Load: the amount of filesystem data under the cassandra data directory and updated every 90 seconds.
- Tokens: number of tokens set for the node.
For more Info: https://docs.datastax.com/en/cassandra/3.0/cassandra/tools/toolsStatus.html
2. MONITOR CASSANDRA CLUSTER SINGLE NODE INFO
If we need to require cluster node stats including:
- Uptime
- Heap Memory
- Exceptions Count
- Key Cache Hit Rate
and more, for complete we need to execute below command. But this command gave us the stats of current node in which we are making ssh.
sudo docker run -t --net=host cassandra:3.0 nodetool info
OUTPUT
ID
Gossip active true Thrift active false Native Transport active true Load 22.57 GB Generation No 1506714058 Uptime (seconds) 1089322 Heap Memory (MB) 1594.28 / 4086.00 Off-Heap Memory (MB) 3.42 Data Center dc1 Rack rac1 Exceptions 0 Key Cache entries 158173, size 56.76 MB, capacity 100 MB, 9092924 hits, 9352093 requests, 0.972 recent hit rate, 14400 save period in seconds Row Cache entries 0, size 0 bytes, capacity 0 bytes, 0 hits, 0 requests, NaN recent hit rate, 0 save period in seconds Counter Cache entries 0, size 0 bytes, capacity 50 MB, 0 hits, 0 requests, NaN recent hit rate, 7200 save period in seconds Token (invoke with -T/–tokens to see all 256 tokens)
For more Info:https://docs.datastax.com/en/cassandra/3.0/cassandra/tools/toolsInfo.html
3. MONITOR CLUSTER STATISTICS OF KEYSPACES AND TABLES
If required to monitor statistics of all keyspaces and tables like:
- Read Latency
- Write Latency
- Total Disk Used
and more. For complete we need to execute below command.
sudo docker run -t --net=host cassandra:3.0 nodetool tablestats // Show stats for all keyspaces and tables sudo docker run -t --net=host cassandra:3.0 nodetool tablestats . … . // show stats for specific keyspace and tables
OUTPUT
Keyspace: Read Count: 75738 Read Latency: 1.0500195410494073 ms. Write Count: 34312 Write Latency: 0.12817600256470038 ms. Pending Flushes: 0 Table: SSTable count: 6 Space used (live): 3390578917 Space used (total): 3390578917 Space used by snapshots (total): 0 Off heap memory used (total): 423687 SSTable Compression Ratio: 1.003568418469647 Number of keys (estimate): 3967 Memtable cell count: 94 Memtable data size: 11470945 Memtable off heap memory used: 0 Memtable switch count: 75 Local read count: 75738 Local read latency: 1.032 ms Local write count: 34312 Local write latency: 6.078 ms Pending flushes: 0 and more...
For more Info: https://docs.datastax.com/en/cassandra/3.0/cassandra/tools/toolsTablestats.html
4. MONITOR CLUSTER DISK SPACE RECLAIM
If we require checking, how many garbage value we have and need to monitor how much disk is available for re-claim, we need to execute below command.
sudo docker run -t --net=host cassandra:3.0 nodetool gcstats
OUTPUT:
Interval (ms) Max GC Elapsed (ms) Total GC Elapsed (ms) Stdev GC Elapsed (ms) GC Reclaimed (MB) Collections Direct Memory Bytes 1091695368 530 1353160 12 4603764734720 56632 -1For more Info: https://docs.datastax.com/en/cassandra/3.0/cassandra/tools/toolsGcstats.html
5. MONITOR CLUSTER THREADS AND PENDING PROCESS
Cassandra is based on a Staged Event Driven Architecture (SEDA). Different tasks are separated into stages that are connected by a messaging service. Stages have a queue and thread pool. Some stages skip the messaging service and queue tasks immediately on a different stage when it exists on the same node. The queues can back up if executing at the next stage is too busy and cause performance bottlenecks.
So, if we require monitoring cluster threads, active task, pending task, completed blocked and more. It’s updated when SSTables change through compaction or flushing. we need to execute below command:
sudo docker run -t --net=host cassandra:3.0 nodetool tpstats
6. MONITOR CLUSTER THREADS AND PENDING PROCESS
If required statistics about the table like:
- Read/Write Latency
- Partition Size
- Column Count
- Number of SSTable
and more… This command is kind of tablestats command, but its output gave us histogram matrices for stats. Below is the command:
EXAMPLE:
sudo docker run -t --net=host cassandra:3.0 nodetool tablehistograms
OUTPUT
Percentile SSTables Write Latency (micros) Read Latency (micros) Partition Size Cell Count 50% 0.00 182.79 0.00 1331 4 75% 1.00 182.79 0.00 1331 4 95% 1.00 219.34 0.00 1597 4 98% 1.00 379.02 0.00 1597 4 99% 1.00 379.02 0.00 1597 4 Min 0.00 51.01 0.00 447 4 Max 1.00 1629.72 0.00 1916 4For more Info: https://docs.datastax.com/en/cassandra/3.0/cassandra/tools/toolsTablehisto.html
As we mention above there are lots of commands are available for monitoring and manage cluster by using nodetool. So, as per our requirements, we will use nodetool for manage or monitor.
References:
https://docs.datastax.com/en/cassandra/3.0/cassandra/tools/toolsNodetool.html
Recommend
About Joyk
Aggregate valuable and interesting links.
Joyk means Joy of geeK