Skip to content

Troubleshooting

Web Interface Errors and Probable Issues

1. The error occurs in the web interface/Case 1

Error: An Internal Error Has Occurred. Please check that the required services are running.

Probably the mysql server is off/crash. Check mysql server using ssh with the following command:

systemctl status mysql.service

If the service does not look like it's running:

mysql.service - LSB: Start and stop the mysql database server daemon
 Loaded: loaded (/etc/init.d/mysql)
 Active: inactive (dead) since Mon 2016-09-12 09:37:28 EEST; 1min 43s ago
 Process: 15510 ExecStop=/etc/init.d/mysql stop (code=exited, status=0/SUCCESS)
 Process: 548 ExecStart=/etc/init.d/mysql start (code=exited, status=0/SUCCESS)

The problem is resolved with the mysql.service restart system, using the following command:

systemctl restart mysql.service

Check using the following command:

systemctl status mysql.service

The result should look like this:

mysql.service - LSB: Start and stop the mysql database server daemon
   Loaded: loaded (/etc/init.d/mysql)
   Active: active (running) since Mon 2016-09-12 09:40:48 EEST; 2s ago
  Process: 15510 ExecStop=/etc/init.d/mysql stop (code=exited, status=0/SUCCESS)
  Process: 15959 ExecStart=/etc/init.d/mysql start (code=exited, status=0/SUCCESS)

2. The error occurs in the web interface/Case 2

Error: Connecting to the main datastore. Are all the services running? Connection refused

Probably the mysql server is off/crash. Check mysql server using ssh with the following command:

systemctl status mysql.service

If the service does not look like it's running :

mysql.service - LSB: Start and stop the mysql database server daemon
   Loaded: loaded (/etc/init.d/mysql)
   Active: inactive (dead) since Mon 2016-09-12 09:37:28 EEST; 1min 43s ago
  Process: 15510 ExecStop=/etc/init.d/mysql stop (code=exited, status=0/SUCCESS)
  Process: 548 ExecStart=/etc/init.d/mysql start (code=exited, status=0/SUCCESS)

The problem is resolved with the mysql.service restart system, using the following command:

systemctl restart mysql.service

Check using the following command:

systemctl status mysql.service

The result should look like this:

mysql.service - LSB: Start and stop the mysql database server daemon
   Loaded: loaded (/etc/init.d/mysql)
   Active: active (running) since Mon 2016-09-12 09:40:48 EEST; 2s ago
  Process: 15510 ExecStop=/etc/init.d/mysql stop (code=exited, status=0/SUCCESS)
  Process: 15959 ExecStart=/etc/init.d/mysql start (code=exited, status=0/SUCCES

Or probably Online DataStorage is off/crash. Check Online DataStorage service using ssh with the following command: 

systemctl status opensearch.service

If the service does not look like it's running :

opensearch.service - opensearch init script wrapper
     Loaded: loaded (/lib/systemd/system/opensearch.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Fri 2023-04-28 13:13:38 EEST; 2 min 5s ago
    Process: 3416812 ExecStart=/bin/bash -c /usr/bin/docker compose -f /var/opt/cyberquest/compose/docker-compose.yml up opensearch > /var/log/opensearch.log (code=exited, status=1/FAILURE)
   Main PID: 3416812 (code=exited, status=1/FAILURE)
        CPU: 124ms

The problem is resolved with the ** opensearch.service restart** system, using the following command:

systemctl restart opensearch.service

Check using the following command:

systemctl status opensearch.service

The result should look like this:

opensearch.service - opensearch init script wrapper
     Loaded: loaded (/lib/systemd/system/opensearch.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2023-05-08 14:58:30 EEST; 23s ago
   Main PID: 52055 (bash)
      Tasks: 19 (limit: 19172)
     Memory: 22.7M
        CPU: 141ms
     CGroup: /system.slice/opensearch.service
             ├─52055 /bin/bash -c /usr/bin/docker compose -f /var/opt/cyberquest/compose/docker-compose.yml up opensearch > /var/log/opensearch.log
             ├─52056 /usr/bin/docker compose -f /var/opt/cyberquest/compose/docker-compose.yml up opensearch
             └─52070 /usr/libexec/docker/cli-plugins/docker-compose compose -f /var/opt/cyberquest/compose/docker-compose.yml up opensearch

3. The error occurs in the web interface/Case 3

Error: Connecting to the main datastore. Are all the services running? Online DataStorage Error: Error

Probably Online DataStorage is off/crash. Check Online DataStorage service using ssh with the following command: 

systemctl status opensearch.service

If the service does not look like it's running:

opensearch.service - opensearch init script wrapper
     Loaded: loaded (/lib/systemd/system/opensearch.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Fri 2023-04-28 13:13:38 EEST; 2 min 5s ago
    Process: 3416812 ExecStart=/bin/bash -c /usr/bin/docker compose -f /var/opt/cyberquest/compose/docker-compose.yml up opensearch > /var/log/opensearch.log (code=exited, status=1/FAILURE)
   Main PID: 3416812 (code=exited, status=1/FAILURE)
        CPU: 124ms

The problem is resolved with the ** opensearch.service restart** system, using the following command:

systemctl restart opensearch.service

Check using the following command:

systemctl status opensearch.service

The result should look like this:

opensearch.service - opensearch init script wrapper
     Loaded: loaded (/lib/systemd/system/opensearch.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2023-05-08 14:58:30 EEST; 23s ago
   Main PID: 52055 (bash)
      Tasks: 19 (limit: 19172)
     Memory: 22.7M
        CPU: 141ms
     CGroup: /system.slice/opensearch.service
             ├─52055 /bin/bash -c /usr/bin/docker compose -f /var/opt/cyberquest/compose/docker-compose.yml up opensearch > /var/log/opensearch.log
             ├─52056 /usr/bin/docker compose -f /var/opt/cyberquest/compose/docker-compose.yml up opensearch
             └─52070 /usr/libexec/docker/cli-plugins/docker-compose compose -f /var/opt/cyberquest/compose/docker-compose.yml up opensearch

4. Mysql service off

The error which appears in the “putty” with the corresponding host name (or ip address) and the port and also the connection type (SSH) using the command:

systemctl status mysql.service

And the error following commands:

Alt Image

Now if we go into the web application this will be the error that will occur:

Error: An Internal Error Has Occurred. Please check the that required services are running

Alt Image

The resolving method is to restart mysql service by following the command:

systemctl restart mysql.service

Check using the following command:

systemctl status mysql.service

And the result of the command is the following:

Alt Image

5. Online DataStorage service off

The error which appears in the “putty” with the corresponding host name (or ip address) and the port and also the connection type (SSH) using the command:

systemctl status opensearch.service

And the error following commands:

Alt Image

Now if we go into cerebro web plugin (http://CyberquestIP:9000) will show the following error:

Alt Image

Or if we go into the web application this will be the error that will occur and there is no data available:

Alt Image

The resolving method is to restart the service by following the command:

systemctl restart opensearch.service

Check using the following command:

systemctl status opensearch.service

And the result of the command is the following:

Alt Image

6. Rabbitmq-server service stopped

The error which appears in the “putty” with the corresponding host name (or ip address) and the port and also the connection type (SSH) using the command:

systemctl status rabbitmq-server.service

And the error following commands:

Alt Image

Now if we go into rabbitmq web plugin (http://CyberquestIP:15672) does not work:

Alt Image

The resolving method is to restart the service by following the commands:

systemctl restart rabbitmq-server.service

AND

systemctl restart data-acquisition.service

Check using the following command:

systemctl status rabbitmq-server.service

And the result of the command is the following:

Alt Image

7. Nginx.service service stopped

The error which appears in the “putty” with the corresponding host name (or ip address) and the port and also the connection type (SSH) using the command:

systemctl status nginx.service

And the error following commands:

Alt Image

Or if we go into the web application this will be the error that will occur:

Alt Image

The resolving method is to restart the service by following the command:

systemctl restart nginx.service

Check using the following command:

systemctl status nginx.service

The result of the command is the following:

Alt Image

8. Php8.1-fpm.service service stopped

The error which appears in the “putty” with the corresponding host name (or ip address) and the port and also the connection type (SSH) using the command:

systemctl status php8.1-fpm.service

And the error following commands:

Alt Image

Or if we go into the web application this will be the error that will occur:

Alt Image

The resolving method is to restart the service by following the command:

systemctl restart php8.1-fpm.service

Check using the following command:

systemctl status php8.1-fpm.service

The result of the command is the following:

Alt Image

9. Data-storage.service service stopped

The error which appears in the “putty” with the corresponding host name (or ip address) and the port and also the connection type (SSH) using the command:

systemctl status data-storage.service

And the error following commands:

Alt Image

Events blocked on the queue in Data Storage on RabbitMQ(http://CyberquestIP:15672)

Alt Image

The resolving method is to restart the service by following the command:

systemctl restart data-storage.service

Check using the following command:

systemctl status data-storage.service

The result of the command is the following:

Alt Image

10. Data-acquisition.service service stopped

The error which appears in the “putty” with the corresponding host name (or ip address) and the port and also the connection type (SSH) using the command:

systemctl status data-acquisition.service

And the error following commands:

Alt Image

Events blocked on the queue in data-acquisition.service on RabbitMQ (http://CyberquestIP:15672)

Alt Image

The resolving method is to restart the service by following the command:

systemctl restart data-acquisition.service

Check using the following command:

systemctl status data-acquisition.service

The result of the command is the following:

Alt Image

Or if we go into the web application this will be the error that will occur:

ERROR: Connecting to the main datastore. Are all the services running? Index “el_logs_current” is missing

Alt Image

Now if we go into cerebro web plugin (http://CyberquestIP:9000) will show the following error:

Alt Image

The resolving method is to restart the service by following the command:

systemctl restart data-acquisition.service

11. Rsyslog service stopped (self-audit events)

The error which appears in the “putty” with the corresponding host name (or ip address) and the port and also the connection type (SSH) using the command:

/etc/init.d/rsyslog status

And the error following commands:

Alt Image

The resolving method is to restart the service by following the command:

/etc/init.d/rsyslog restart

And the result of the command is the following:

/etc/init.d/rsyslog status

Alt Image

Or another problem can be found in rsyslog.conf file: 

nano /etc/rsyslog.conf

Scroll down until you reach the end of the file and check if it matches the desired ip (" *. * " - this symbol represents all types of events)

Alt Image

12. RabbitMQ error

When this error occurs, we don’t find new information in the web application. We find this error in CYBERQUEST logs. Using the baretail program we open the file(agent.txt) and we can view the error log.

Alt Image

To solve the problem we have to stop the CYBERQUEST Agent and restart the RabbitMQ service. Follow the instructions for more details:

A. Stop CYBERQUEST Agent

On the Windows machine where the CYBERQUEST Agent is installed, open Windows Services and stop the CYBERQUEST Agent service.

Alt Image

To check if the CYBERQUEST Agent has stopped, open Task Manager > Details and wait until the Agent .exe process disappears from Task Manager.

Alt Image

B. Restart RabbitMQ

We connect to the CYBERQUEST server via ssh (e.g. ssh user@ip) and restart RabbitMQ using the following command:

systemctl restart rabbitmq-server.service

C. Start CYBERQUEST Agent

On the Windows machine where the CYBERQUEST Agent is installed, open Windows Services and start the CYBERQUEST Agent service.

Alt Image

To verify that the CYBERQUEST Agent has started, open Task Manager> Details and wait until the Agent.exe process appears in the Task Manager.