Backlog Enterprise Operations Guide

Directory structure

Backlog is installed in the following configurations:

<Installation directory>
├── backlog-job-worker
│     └── hikaricp.properties #Maximum number of connections for database access
├── data #Store location of each data
├── logs #Store location of each log file
├── .env #File of each setting value.
├── docker-compose.yml #Definition file of each container
├── Licenses.txt #Information about open source software being used
└── self-signed-cert #Storage location of internal SSL certificates

Command line of start/stop on Backlog

Stop

Stop running Backlog.

cd <Installation directory>
docker-compose down

Start

Start Backlog.

cd <Installation directory>
docker-compose up -d

Services

Port numbers

Services are published using the following port numbers. If you’re using iptables or other software, please allow external access to these numbers.

File data storage directory

Backlog file data is stored in the following directories:

data
├── attachment
│       └── pull_request
├── davsvn
│       ├── share
│       └── svn
├── elasticsearch
├── git
├── image
└── solr
        ├── issue
        ├── pull_request
        ├── shared_file
        └── wiki

Log file storage directory

Logs for each service are stored in <Installation directory>/logs.
See details on services.

logs
├── ${tag[1]} #Directory for temporary stores logs
│       └── ${tag[1]}
├── backlog-api
├── backlog-davsvn
├── backlog-davsvn-mntlog
│       ├── dav-job-worker
│       ├── httpd-davsvn
│       ├── svn-hook
│       └── svnserve
├── backlog-job-worker
├── backlog-solr
├── backlog-web
├── database-migration
├── elasticsearch
├── elasticsearch-init
├── git-http
├── git-rpc
├── git-rpc-mntlog
├── git-ssh
├── kanban-backend
├── kanban-notification
├── memcached
├── nginx
├── redis
├── update-tool
└── dewey

Parameters for .env

Back-up data

The data handled by Backlog is stored in the following files, which should be backed up regularly.

• Database
• Saved file data directory
• Saved log file directory

Creating an advanced configuration file

Create docker-compose.override.yml on the same level asdocker-compose.yml. In docker-compose.override.yml, list the services you want to change and their configuration values. After it’s created, restart with the Start command.

For example:

services:
  backlog-web:
    environment:
    ...

  backlog-api:
    environment:
    ...

Get more information.

Integrating with Active Directory

Change the port number of LDAPS

For an LDAPS connection, enter the LDAPS port number in the advanced configuration file you created. After it’s created, restart with the Start command.

backlog-api:
  environment:
    - LDAPS_PORT=8636 # Default 636

backlog-web:
  environment:
    - LDAPS_PORT=8636 # Default 636

Use Active Directory Certificate Service

If you use an LDAPS connection and the Active Directory Certificate Service, place the certificate issued by Active Directory Certificate Service in any directory. After it’s placed, add the following information to the advanced configuration file you created, and restart with the Start command.

backlog-api:
  volumes:
    - /path/to/cert/dir:/mnt/certs #Please replace `/path/to/cert/dir` with the directory where you placed the certificate
  command: -Dcom.sun.jndi.ldap.object.disableEndpointIdentification=true

backlog-web:
  volumes:
    - /path/to/cert/dir:/mnt/certs #Please replace `/path/to/cert/dir` with the directory where you placed the certificate
  command: -Dcom.sun.jndi.ldap.object.disableEndpointIdentification=true

Create users with Active Directory

Create Backlog users with information from Active Directory. Set up zone definitions and SRV resource records for Active Directory before operation. Active Directory is supported on Windows Server 2012 R2, Windows Server 2016, and Windows Server 2019. Get detailed instructions.

Regenerate the search index

If the results of a keyword search are not displayed correctly, run the command to regenerate the search indexing. Note that playback is delayed if there’s a large amount of index data.

Problem with issue search (issue list)

Use the following:

docker-compose up elasticsearch-tool

Problem with global navigation keyword searches

Check if the following message is output in the backlog-solr log within the log storage directory:

java.lang.OutOfMemoryError: Java heap space

If the message appears in the log, there might be insufficient memory for Solr. Open the docker-compose.override.yml file and update the Solr memory settings.

backlog-solr:
  environment:
    - SOLR_JAVA_MEM=-Xms1024m -Xmx1024m # Default -Xms512m -Xmx512m

With the memory settings updated, restart using the same command you used to start.
After restarting, execute the following. The search index will be recreated.

docker-compose up solr-tool

If the message doesn’t appear in the log, use the following:

docker-compose up solr-tool

Change the speed of search index re-generation

Change the limits for indexing jobs in the advanced configuration file you created. After editing, restart with the Start command.

backlog-job-worker:
  environment:
    - BACKLOG_WORKER_JOB_INDEXING_ISSUE_JOB_PERMITS=16 # Default 6

Change the maximum of bucket size requests

If the wiki has too many characters and can’t be saved, for example, change the bucket size limit in the advanced configuration file you created. After editing, restart with the Start command.

backlog-api:
  environment:
    - MAX_MEMORY_BUFFER=25MB # Default 20MB
    - MAX_DISK_BUFFER=60MB # Default 55MB

backlog-web:
  environment:
    - MAX_MEMORY_BUFFER=25MB
    - MAX_DISK_BUFFER=60MB

Change the maximum number of simultaneous Nginx connections

Copy nginx.conf locally with the following command.

docker-compose cp nginx:/etc/nginx/nginx.conf .

Edit nginx.conf and docker-compose.override.yml.

nginx:
  volumes:
    - ./nginx.conf:/etc/nginx/nginx.conf

Get more information on configuring nginx.conf.

After editing, remove and restart Nginx using the following command.

docker-compose rm -fs nginx
docker-compose up -d nginx

Notes on using MySQL 5.7

Troubleshooting

MySQL 5.7 may cause issues in Backlog if the MySQL query cache is enabled. If query_cache_type has an existing value in my.cnf, set it to 0. If not, there’s no need to add it as the default value for query_cache_type in MySQL 5.7 is 0.

[mysqld]
max_allowed_packet=128M
character-set-server=utf8mb4
sql_mode = "NO_ENGINE_SUBSTITUTION"
query_cache_type = 0

[mysql]
default-character-set=utf8mb4

[client]
default-character-set=utf8mb4

Unable to attach large files to issues

You can attach files up to 50MB to an issue. If you have any problems, try adjusting the following settings:

Increase the request packet size limit

If MAX_DISK_BUFFER is set to less than 55MB in the advanced configuration file, please either remove the setting or change it to at least 55MB. Then, restart using the Start command.

backlog-api:
  environment:
    - MAX_DISK_BUFFER=55MB #default 55MB

backlog-web:
  environment:
    - MAX_DISK_BUFFER=55MB

Modify or add MySQL settings

In the my.cnf file, set the value of max_allowed_packet to 128M or higher. This is required for MySQL 8.4, MySQL 8.0, and MySQL 5.7. After editing, restart MySQL.

[mysqld]
max_allowed_packet=128M

Remove unnecessary Docker images

To batch delete Docker images from a specific version, use the following command. Here’s an example of removing all images with version 2.0.0:

docker images | grep '019623933850.dkr.ecr.ap-northeast-1.amazonaws.com' | awk '{print $1 ":2.0.0"}' | xargs docker rmi

Connecting to the database using SSL

This section explains how to configure SSL encryption for your connection to MySQL. The steps vary depending on the type of certificate you are using (public CA or private CA).

Using a public Certificate Authority (CA) certificate

These are the configuration steps when using a certificate from a public Certificate Authority (e.g., DigiCert, Let’s Encrypt).

Prerequisites

• You must be using MySQL 8.4.
• SSL connections must be enabled on the MySQL server.

Configuration steps

Step 1: Add MySQL SSL connection settings to .env

Modify the items in the .env file as follows:

DB_USE_SSL=true
JDBC_OPTIONS='&useSsl=true'

Step 2: Edit the advanced configuration file

• Load the contents of the .env file into the advanced configuration file.
• After editing, restart with the Startcommand.

services:
  database-migration:
    environment:
      - FLYWAY_URL=jdbc:mysql://$BACKLOG_DB_HOST:$BACKLOG_DB_PORT/$BACKLOG_DB_NAME?allowPublicKeyRetrieval=true${JDBC_OPTIONS}
  backlog-api:
    environment:
      - JDBC_OPTIONS=${JDBC_OPTIONS}
  backlog-web:
    environment:
      - JDBC_OPTIONS=${JDBC_OPTIONS}
  backlog-job-worker:
    environment:
      - JDBC_OPTIONS=${JDBC_OPTIONS}
  elasticsearch-init:
    environment:
      - JDBC_OPTIONS=${JDBC_OPTIONS}
  elasticsearch-tool:
    environment:
      - JDBC_OPTIONS=${JDBC_OPTIONS}
  update-tool:
    environment:
      - JDBC_OPTIONS=${JDBC_OPTIONS}
  kanban-backend:
    environment:
      - MYSQL_JDBC_OPTIONS=${JDBC_OPTIONS}
  kanban-notification:
    environment:
      - BACKLOG_NOTIFICATIONS_DB_USE_SSL=${DB_USE_SSL}
  backlog-davsvn:
    environment:
      - BACKLOG_DB_USE_SSL=${DB_USE_SSL}
  git-ssh:
    environment:
      - DB_USE_SSL=${DB_USE_SSL}
  git-http:
    environment:
      - DB_USE_SSL=${DB_USE_SSL}
  git-rpc:
    environment:
      - BACKLOG_DB_USE_SSL=${DB_USE_SSL}
  git-backlog-worker:
    environment:
      - BACKLOG_DB_USE_SSL=${DB_USE_SSL}
  git-webhook-worker:
    environment:
      - BACKLOG_DB_USE_SSL=${DB_USE_SSL}
  backlog-solr:
    environment:
      - SOLR_OPTS=-Denable.packages=true -Ddataimporter.request.db_url=jdbc:mysql://$BACKLOG_DB_HOST:$BACKLOG_DB_PORT/$BACKLOG_DB_NAME?useUnicode=true&characterEncoding=UTF-8&useServerPrepStmts=false&allowPublicKeyRetrieval=true${JDBC_OPTIONS} -Ddataimporter.request.db_user=$BACKLOG_DB_USER -Ddataimporter.request.db_pass=$BACKLOG_DB_PASSWORD

Using a self-signed or private Certificate Authority (CA) certificate

These are the configuration steps when using a self-signed certificate or a certificate issued by a private CA. You need to explicitly load the trusted CA certificate into Backlog.

Prerequisites

• You must be using MySQL 8.4.
• SSL connections must be enabled on the MySQL server.
• The CA certificate file must be located on the Backlog server.
  • Do not place anything other than the CA certificate, such as unrelated files or private keys, in the directory where the CA certificate is located.

Configuration steps

Step 1: Add MySQL SSL connection settings to .env

Modify the following in the .env file:
Change the CA_certificate_name placeholder to appropriately match your actual file name.
Do not edit the /mnt/mysqlcerts part as it represents a path inside the Docker container; edit only the CA certificate name.

DB_USE_SSL=true
DB_SSL_CA_PATH='/mnt/mysqlcerts/CA_certificate_name'
JDBC_OPTIONS='&useSsl=true&disableSslHostnameVerification=true&serverSslCert=/mnt/mysqlcerts/CA_certificate_name'

Step 2: Edit the advanced configuration file

Load the contents of the .env file into the advanced configuration file and:

• Mount the directory containing the CA certificate to a path inside the container.
• Replace the /path/to/cert/dir part with the actual directory path where you have placed the CA certificate.
• After editing, restart with the Startcommand.

services:
  database-migration:
    environment:
      - FLYWAY_URL=jdbc:mysql://$BACKLOG_DB_HOST:$BACKLOG_DB_PORT/$BACKLOG_DB_NAME?allowPublicKeyRetrieval=true${JDBC_OPTIONS}
    volumes:
      - /path/to/cert/dir:/mnt/mysqlcerts
  backlog-api:
    environment:
      - JDBC_OPTIONS=${JDBC_OPTIONS}
    volumes:
      - /path/to/cert/dir:/mnt/mysqlcerts
  backlog-web:
    environment:
      - JDBC_OPTIONS=${JDBC_OPTIONS}
    volumes:
      - /path/to/cert/dir:/mnt/mysqlcerts
  backlog-job-worker:
    environment:
      - JDBC_OPTIONS=${JDBC_OPTIONS}
    volumes:
      - /path/to/cert/dir:/mnt/mysqlcerts
  elasticsearch-init:
    environment:
      - JDBC_OPTIONS=${JDBC_OPTIONS}
    volumes:
      - /path/to/cert/dir:/mnt/mysqlcerts
  elasticsearch-tool:
    environment:
      - JDBC_OPTIONS=${JDBC_OPTIONS}
    volumes:
      - /path/to/cert/dir:/mnt/mysqlcerts
  update-tool:
    environment:
      - JDBC_OPTIONS=${JDBC_OPTIONS}
    volumes:
      - /path/to/cert/dir:/mnt/mysqlcerts
  kanban-backend:
    environment:
      - MYSQL_JDBC_OPTIONS=${JDBC_OPTIONS}
    volumes:
      - /path/to/cert/dir:/mnt/mysqlcerts
  kanban-notification:
    environment:
      - BACKLOG_NOTIFICATIONS_DB_USE_SSL=${DB_USE_SSL}
      - BACKLOG_NOTIFICATIONS_DB_SSL_CA_PATH=${DB_SSL_CA_PATH}
    volumes:
      - /path/to/cert/dir:/mnt/mysqlcerts
  backlog-davsvn:
    environment:
      - BACKLOG_DB_USE_SSL=${DB_USE_SSL}
      - BACKLOG_DB_SSL_CA_FILE=${DB_SSL_CA_PATH}
    volumes:
      - /path/to/cert/dir:/mnt/mysqlcerts
  git-ssh:
    environment:
      - DB_USE_SSL=${DB_USE_SSL}
      - DB_CA_PATH=${DB_SSL_CA_PATH}
    volumes:
      - /path/to/cert/dir:/mnt/mysqlcerts
  git-http:
    environment:
      - DB_USE_SSL=${DB_USE_SSL}
      - DB_CA_PATH=${DB_SSL_CA_PATH}
    volumes:
      - /path/to/cert/dir:/mnt/mysqlcerts
  git-rpc:
    environment:
      - BACKLOG_DB_USE_SSL=${DB_USE_SSL}
      - BACKLOG_DB_SSL_CA_FILE=${DB_SSL_CA_PATH}
    volumes:
      - /path/to/cert/dir:/mnt/mysqlcerts
  git-backlog-worker:
    environment:
      - BACKLOG_DB_USE_SSL=${DB_USE_SSL}
      - BACKLOG_DB_SSL_CA_FILE=${DB_SSL_CA_PATH}
    volumes:
      - /path/to/cert/dir:/mnt/mysqlcerts
  git-webhook-worker:
    environment:
      - BACKLOG_DB_USE_SSL=${DB_USE_SSL}
      - BACKLOG_DB_SSL_CA_FILE=${DB_SSL_CA_PATH}
    volumes:
      - /path/to/cert/dir:/mnt/mysqlcerts
  backlog-solr:
    environment:
      - SOLR_OPTS=-Denable.packages=true -Ddataimporter.request.db_url=jdbc:mysql://$BACKLOG_DB_HOST:$BACKLOG_DB_PORT/$BACKLOG_DB_NAME?useUnicode=true&characterEncoding=UTF-8&useServerPrepStmts=false&allowPublicKeyRetrieval=true${JDBC_OPTIONS} -Ddataimporter.request.db_user=$BACKLOG_DB_USER -Ddataimporter.request.db_pass=$BACKLOG_DB_PASSWORD
    volumes:
      - /path/to/cert/dir:/mnt/mysqlcerts

About the trademark

• MySQL and Java are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective owners.
• Docker are trademarks or registered trademarks of Docker, Inc. in the United States and/or other countries.
• Nginx are registered trademarks of Nginx Software Inc. in the United States and/or other countries.
• Active Directory is either a registered trademark or a trademark of Microsoft Corporation in the United States and/or other countries.
• Other company and product names in this document may be trademarks or registered trademarks of their respective owners.