Stopping nodetool repair in Cassandra

If you see multiple repair process, when you give nodetool tpstats implies that repair process was running may be initiated by nodetool with full and/or nodetool -pr and/or datastax-opscenter(if initiated in opscenter).

Usually repairs will be failed by throwing like "validation failed" so there is no way to directly stop the repairs if it still showing in nodetool tpstats

This can be stopped by, performed by calling forceTerminateAllRepairSessions in the StorageServiceMBean via JMX. From there you would need to run the "nodetool stop VALIDATION" on each node to kill the validation compactions.

In system.log file if it shows "primary range: true" implies repair is started with -pr and if system.log shows  primary range: false implies repair is started with -full option.

***

Delete multiple rows form cassandra

cqlsh 
> copy bypramod(username, id, random, columns) to '/tmp/tp.csv' WITH DELIMITER ='\t';


Shell 
grep ^PRIMARYKEY$'\t' tp.csv | awk -F'\t' '{print "delete from examplekeyspace.bypramod where username='\''" $1 "'\'' and id='\''" $2 "'\'' and random=" $3 ";"}' >secondfile.cql

On new keyspace import the file.

cqlsh
source 'seconfile.cql'

***

File descriptor: To list open files in a machine

lsof -p <pid>

Lists open files for the process.

lsof | grep ‘(deleted)’

Lists the deleted open files

lsof -a +L1 *mountpoint*

Lists open files for the mount point (-a => and) (1=> below values which shows zero => unlinked)

Losf -I

Lists all open internet socket files

***

Verifying the interface speed configuration

To verify the max speed for an interface, use the below command which will show the speed.

ethtool eth0

To modify the speed of the interface use below command

ethtool --speed 1000 eth0

**Number above is in Mb/s

***

Other Method:

Install iperf on two nodes using apt-get or yum.

On the first node run, 

command: iperf -s -f M

Then on the second node

command: iperf -c x.x.x.x -f M

Here x.x.x.x is the ip address of first node where the iperf is running. This will show a test of transfer speed between the two nodes.

***

Turning off/ disabling swap memory

To turn of or disable the swap memory,

command: swapoff -a

Above command turns off the swap until reboot of machine, to make changes persistent comment out swap  in /etc/fstab file

Verify: free -h

***

Install Java 8 in CentOS 7

Installing Java 8 in CentOS.

sudo yum install java-1.8.0-openjdk

Verify: java -version

If you want tools like jstatd etc devel tools, you need to install

sudo yum install java-1.8.0-openjdk-devel

Verify: rpm -qa | grep java

For monitoring GC by jvisualvm

Create jstatd policy file by:

vi /tmp/jstatd.all.policy

grant codebase "file:${java.home}/../lib/tools.jar" {
   permission java.security.AllPermission;
};

Start the process by, jstatd -J-Djava.security.policy=/tmp/jstatd.all.policy

**In above policy file, find java.home easily by

find / -name tools.jar
example output: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.151-5.b12.el7_4.x86_64/lib/tools.jar

Update the complete path from the above output

***

Garbage Collection in Cassandra

CONCURRENT MARK SWEEP (CMS) COLLECTOR:

CMS is like serial process where memory is allocated in such a way that eden, survivor, old/tenured and permanent generations. And the objects need to be moved from one generation to other verifying the lively ness.

Its a low pause collector, imples it collects tenured generation, minimize the pauses due to garbage collection by doing most collection by doing most collection work concurrently with the application threads. For young generation uses same as parallel collector.

GARBAGE FIRST (G1GC)

In G1GC, the complete heap is divided in to chunks (~2000) where collector is parallel, concurrent and incrementally compacting low pause garbage collector.
Old generation: Single threaded

Compaction: Multi threaded for young and single threaded for old

Flags: -XX:+UseParallelGC

Young generation: multithreaded (Choose less than or equal to number of CPU cores, not include hyperthreads, for number of core check by command "lscpu" = Core(s) per socket * Socket(s) )

Flags: 

JVM_OPTS="$JVM_OPTS -XX:ParallelGCThreads=[Core(s) per socket * Socket(s)]"

JVM_OPTS="$JVM_OPTS -XX:ConcGCThreads=[1/4 th of (Core(s) per socket * Socket(s))]"

Enable Parallel referencing:
JVM_OPTS="$JVM_OPTS -XX:+ParallelRefProcEnabled"

Delay region scanning: Java 8 by default is 40%
JVM_OPTS="$JVM_OPTS -XX:InitiatingHeapOccupancyPercent=70"

Enable GC logging: 

JVM_OPTS="$JVM_OPTS -XX:+PrintGCDetails"

JVM_OPTS="$JVM_OPTS -XX:+PrintGCDateStamps"

JVM_OPTS="$JVM_OPTS -XX:+PrintHeapAtGC"

JVM_OPTS="$JVM_OPTS -XX:+PrintTenuringDistribution"

JVM_OPTS="$JVM_OPTS -XX:+PrintGCApplicationStoppedTime"

JVM_OPTS="$JVM_OPTS -XX:+PrintPromotionFailure"

JVM_OPTS="$JVM_OPTS -XX:PrintFLSStatistics=1"

JVM_OPTS="$JVM_OPTS -Xloggc:/var/log/cassandra/gc-`date +%s`.log"

Flags: -XX:+UseParallelOldGC

This flag implies, all generations are multi threaded like young, old and for compaction also.

For monitoring your GC log easily and threads, try these websites.
1) www.gceasy.io
2) www.fastthread.io

***