7. Performance Tuning¶

7.1. Rationale¶

JProfiler
MAT (Memory Analyzer Tool) [heapdump and MAT from Eclipse]
Performance SQL
own database indexes
pgpool and database cache
nginx as a SSL terminator
Varnish caching REST responses (JSON) and static files
Java Melody
New Relic

7.2. Cache¶

# ACL - office + internal + cloud
acl local_access {
  "localhost";
  "10.0.0.0"/8;
  "172.16.0.0"/12;
}

backend default {
  .host = "127.0.0.1";
  .port = "8090";
  .connect_timeout = 600s;
  .first_byte_timeout = 600s;
  .between_bytes_timeout = 600s;
  .max_connections = 150;
}

sub vcl_recv {

  if (req.request == "PURGE" && !client.ip ~ local_access) {
    error 405 "Not allowed.";
  }

  if (req.request == "REFRESH" && client.ip ~ local_access) {
    set req.hash_always_miss = true;
  }

  if (req.url ~ "\.(swf|ico|css|js)$" && req.url !~ "(secure|rest)") {
    return (lookup);
  }

  if (req.url ~ "\.(xml)"){
    return(pipe);
  }

  if (req.url ~ "^/secure/projectavatar" || req.url ~ "^/secure/useravatar") {
    # set beresp.http.Cache-Control = "max-age=330";
    # set beresp.ttl = 330s;
    return (lookup);
  }


  if (req.request == "POST") {
    return (pipe);
  }

  return (pass);
}

sub vcl_fetch {
  unset beresp.http.Server;
  set beresp.http.X-Backend = beresp.backend.name;


   if (req.url ~ "^/secure/projectavatar"|| req.url ~ "^/secure/useravatar") {
    set beresp.http.Cache-Control = "max-age=330";
    set beresp.ttl = 1d;
    return (deliver);
  }

  if (beresp.http.Content-Type ~ "text" && beresp.http.Content-Encoding != "gzip") {
    set beresp.do_gzip = true;
  }

  if (beresp.http.Content-Type ~ "application/json" && beresp.http.Content-Encoding != "gzip") {
    set beresp.do_gzip = true;
  }

  if (beresp.http.Content-Type ~ "application/javascript" && beresp.http.Content-Encoding != "gzip") {
    set beresp.do_gzip = true;
  }

  # if (beresp.http.Pragma ~ "no-cache" || beresp.http.Cache-Control ~ "no-cache" || beresp.http.Cache-Control ~ "private") {
  #   return(hit_for_pass);
  # }

  if (beresp.http.Cache-Control ~ "max-age" || beresp.http.Cache-Control ~ "s-maxage") {
    unset beresp.http.Set-Cookie;
    unset beresp.http.X-AREQUESTID;
    unset beresp.http.X-ASESSIONID;
    unset beresp.http.X-AUSERNAME;
    unset beresp.http.X-Seraph-LoginReason;
    return(deliver);
  }

  if (beresp.status == 404) {
    set beresp.http.Cache-Control = "max-age=5";
    set beresp.ttl = 5s;
    set beresp.grace = 5s;
  }
}

sub vcl_hit {
  if (req.request == "PURGE") {
    purge;
    error 200 "Cache purged";
  }
}

sub vcl_miss {
  if (req.request == "PURGE") {
    error 404 "Not in cache";
  }
}

sub vcl_deliver {
  # remote varnish and nginx headers
  remove resp.http.X-Varnish;
  remove resp.http.Via;
  remove resp.http.X-Powered-By;
  remove resp.http.Server;
  set resp.http.X-Hit = "HIT " + obj.hits;
  set resp.http.X-Origin = server.hostname;
}

sub vcl_hash {
  hash_data(req.url);
  return (hash);
}

sub vcl_pipe {
  /* Force the connection to be closed afterwards so subsequent reqs don't use pipe */
  set bereq.http.connection = "close";
}

sub vcl_error {
  synthetic "<html><body><!-- Mediation error --></body></html>";
  return (deliver);
}

7.3. Optymalizacje¶

Wyłączyć Activity Stream
Update gadżetów na Dashboardzie (update na bazie dla wszystkich gadgetów)
Edukacja użytkowników aby nie mieli odpalonych miliona zakładek z JIRĄ
Czy wszystkie monitory z Wallboardami są potrzebne?

7.4. Database¶

/var/atlassian/application-data/jira/dbconfig.xml

<pool-min-size>20</pool-min-size>
<pool-max-size>20</pool-max-size>
<pool-max-wait>30000</pool-max-wait>
<validation-query>select 1</validation-query>
<min-evictable-idle-time-millis>60000</min-evictable-idle-time-millis>
<time-between-eviction-runs-millis>300000</time-between-eviction-runs-millis>
<pool-max-idle>20</pool-max-idle>
<pool-remove-abandoned>true</pool-remove-abandoned>
<pool-remove-abandoned-timeout>300</pool-remove-abandoned-timeout>
<pool-test-on-borrow>false</pool-test-on-borrow>
<pool-test-while-idle>true</pool-test-while-idle>

7.5. Garbage Collector¶

Jakub Kubryński on Garbage Collector https://www.youtube.com/watch?v=LCr3XyHdaZk
G1 GC -XX:+UseG1GC
Xmx
/opt/atlassian/jira/bin/setenv.sh

Code 7.1. Jira Garbage Collector¶

JIRA_HOME="/opt/jira/home"
JVM_SUPPORT_RECOMMENDED_ARGS="-server -XX:MaxPermSize=512m -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+OptimizeStringConcat -XX:+PrintGCDetails -XX:+DisableExplicitGC -Xloggc:/opt/jira/logs/gc-jira-$(hostname)-$(date +%Y.%m.%d).log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=10M"
JVM_MINIMUM_MEMORY="512m"
JVM_MAXIMUM_MEMORY="2048m"

# -server
# -XX:MaxPermSize=512m
# -XX:+UseG1GC
# -XX:+PrintGC
# -XX:MaxGCPauseMillis=200
# -XX:+PrintGCDateStamps
# -XX:+PrintGCDetails
# -XX:+UseGCLogFileRotation
# -XX:GCLogFileSize=10M
# -Xloggc:/opt/jira/logs/gc-jira-$(hostname)-$(date +%F).log
# -XX:NumberOfGCLogFiles=10
# -XX:+OptimizeStringConcat
# -XX:+DisableExplicitGC



# -Xms --> Minimum Memory
# -Xmx --> Maximum Memory
# -Xmn --> Heap of Younger Generation
# -Xss --> Thread Stack Size
# -XX:MaxMetaspaceSize --> Maximum Memory for Non-Heap Metaspace.
# -XX:NewRatio --> Ratio between Younger and Older Generation Memory sizes.
# -XX:ParallelGCThreads --> No of Parallel GC threads. By default, the GC threads will be equal to the number of CPUs of the Node / VM. Used when Parallel Garbage collectors are configured.






GC_JVM_PARAMETERS=""
GC_JVM_PARAMETERS="-XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+PrintGCCause ${GC_JVM_PARAMETERS}"
GC_JVM_PARAMETERS="-Xloggc:$LOGBASEABS/logs/atlassian-jira-gc-%t.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=20M ${GC_JVM_PARAMETERS}"




## Defaultowe ustawienia Jiry po instalacji:
/opt/atlassian/jira/jre//bin/java
	-Djava.util.logging.config.file=/opt/atlassian/jira/conf/logging.properties
	-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
	-Xms384m
	-Xmx768m
	-Djava.awt.headless=true
	-Datlassian.standalone=JIRA
	-Dorg.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER=true
	-Dmail.mime.decodeparameters=true
	-Dorg.dom4j.factory=com.atlassian.core.xml.InterningDocumentFactory
	-XX:-OmitStackTraceInFastThrow
	-Datlassian.plugins.startup.options=
	-Djdk.tls.ephemeralDHKeySize=2048
	-Djava.protocol.handler.pkgs=org.apache.catalina.webresources
	-Xloggc:/opt/atlassian/jira/logs/atlassian-jira-gc-%t.log
	-XX:+UseGCLogFileRotation
	-XX:NumberOfGCLogFiles=5
	-XX:GCLogFileSize=20M
	-XX:+PrintGCDetails
	-XX:+PrintGCDateStamps
	-XX:+PrintGCTimeStamps
	-XX:+PrintGCCause
	-classpath /opt/atlassian/jira/bin/bootstrap.jar:/opt/atlassian/jira/bin/tomcat-juli.jar
	-Dcatalina.base=/opt/atlassian/jira
	-Dcatalina.home=/opt/atlassian/jira
	-Djava.io.tmpdir=/opt/atlassian/jira/temp
	org.apache.catalina.startup.Bootstrap start

7.6. Monitorowanie¶

http://www.stagemonitor.org/
New Relic
JavaMelody
JIRA embedded tools (in settings):
- JMX monitoring
- SQL profiling

7.7. Rozwiązywanie problemów¶

grep '/rest' /opt/atlassian/jira/logs/access_log.* |awk '{print $7}' |sort |uniq -c |sort -n

Dużo zapytań API (varnish requestów, np. dashboardów)
Inne usługi wysycające pamięć na maszynie, aż do limitów JAVY
Przy port forwardnig ssh -L 5432:localhost:5432 root@adresIP w /var/lib/pgsql/data/pg_hba.conf musi być md5 przy IPv4 i IPv6
Create issue by URL: http://localhost:8080/secure/CreateIssueDetails!init.jspa?pid=10000&issuetype=10002

7.8. Assignments¶

7.8.1. Administracja - Garbage Collector¶

Zmień Garbage Collector na G1
Zmień Xmx na 1GB
Wepnij Java Melody do monitorowania

7.8.2. Administracja - Zmiana Javy¶

Zainstaluj nową Javę na serwerze w katalogu /opt/java/$VERSION
Utwórz symlink /opt/java/default/ wskazujący na /opt/java/$VERSION (dlaczego to dobra praktyka?)
Zrestartuj Jirę by wykorzystywała nową Javę