Non default, can be configured to log multiple types of operations, such as authentication successes or failures, process executions, file accesses, user commands executed in a TTY, etc.
Each record / log entry contain a msg field, composed of a timestamp and an unique ID. Multiple records generated as part of the same Auditd event can share the same msg field. For example, cat /etc/passwd can generate SYSCALL + EXECVE records for the execution of cat and a PATH record for the access to the /etc/passwd file.
The type field contains the type of the record:
- User authentication and access: USER_LOGIN_SUCCESS, USER_LOGIN_FAILED, USER_AUTH_SUCCESS, USER_AUTH_FAILED, USER_START_SUCCESS, USER_START_FAILED, SESSION_TERMINATED.
- Process execution: EXECVE and SYSCALL.
- Filesystem access: PATH (for relative or absolute file access), CWD (current working directory, useful to reconstruct full path if a relative path has been recorded in PATH records) and OPENAT.
- Commands entered in a TTY console: TTY or by users: USER_CMD.
- Full command-line of process: PROCTITLE. The associated proctitle field MAY be encoded in hexadecimal.
- Network socket connections: SOCKADDR. The associated saddr field contains IP and port information, and can be interpreted directly at event generation (if log_format = ENRICHED is set), or with ausearch -i or simple scripting.
- Account activity: ADD_USER or ADD_GROUP.
- More record types are listed in the RedHat documentation.
If present, the auid field defines the ID of the user upon login and remains the same even if the user's identity changes (for instance with su).
If present, uid / gid and euid / egid fields define the user / group IDs and the effective user / group IDs of the audited process.
If present, the tty and ses fields define respectively the terminal and session from which the audited process was invoked.
For SYSCALL records, the aX field(s) define the arguments / parameters of the syscall, represented by unsigned long long integers and as such cannot be used to determine the values taken by the arguments.
Configuration file notably defining the path of the log files:
/etc/auditd.conf
Configuration defining the rules to apply:
/etc/audit/audit.rules
Rules best practice: https://github.com/Neo23x0/auditd
Current log files (default location):
/var/log/audit/audit.log/var/log/audit/audit.log.1
Rotated log archives (default location):
/var/log/audit/audit.log.*.gz
The aureport and ausearch utilities can (and if possible should) be used to search the auditd log files.
Example:
aureport -i [--login | --executable | ...] [--summary] -if <AUDIT_LOG_FILE>
System information
Name
Type
Description
Information / interpretation
Location
Tool(s)
alternatives logs
System information
Logs of the update-alternatives utility, used to manage alternatives (i.e symbolic links to a given command).
/var/log/alternatives.log
Environment variables information
System information.
Contains system-wide or user scoped persistent environment variables.
-
System-wide configuration file:
/etc/environment
Initialization scripts can also be used to define system-wide or user scoped environment variables.
-
Hostname information
System information
Contains the hostname of the system.
-
/etc/hostname
-
Mounted filesystems information
System information.
Contains information on the mounted file systems, such as partition types (ext3 / ext4, etc.).
Configuration:
/etc/fstab
Mount logs (such as Mounting operation / keyword):
/var/log/dmesg
-
Timezone information
System information
Contains the timezone of the system.
-
/etc/timezone/etc/adjtime/etc/localtime
-
Syslog daemon configuration
System information
The Syslog deamon configuration file(s) notably define where the messages / events received by the Syslog daemon will be outputted. The messages are usually written as plaintext files under /var/log/ but can also be sent over the network.
Example of a configuration file writing logs to common files:
auth,authpriv.* /var/log/auth.log
*.*;auth,authpriv.none -/var/log/syslog
kern.* -/var/log/kern.log
mail.* -/var/log/mail.log
Contrary to NTFS partitions, file information is not stored in a specific location (such as the MFT) for ext* partitions but scattered across blocks or groups of blocks (contiguous blocks combined) across the partition. Blocks have a fixed size, determined at the time the filesystem is created.
On ext4, each block group notably store:
The Super block, only replicated at the beginning of a fraction of block groups however, which contains various information about the file system: block size, location of the inode tables, size of block groups, etc.
The inode bitmap and a data block bitmap that limit the number of inodes and data blocks of that particular block group.
An inode table which is a linear array of inodes (first to last inode) of that particular block group.
Inodes are data struct that define a file and each file is associated with one inode. An inode stores metadata about the file (size, owner UID / GID, permissions, timestamps, etc.) and (direct or indirect) pointers to data blocks that contain the file data, but does not store the file name and file data.
Logs of apt-get / apt operations, including packets installation.
Current log file:
/var/log/apt/history.log
Rotated log archives:
/var/log/apt/history.log.*.gz
aptitude logs
Software installation.
Logs of the aptitude utility (front-end to apt) operations, including packets installation.
/var/log/aptitude
dpkg logs
Software installation.
Logs of dpkg operations, including packets installed / removed through the utility.
Current log files:
/var/log/dpkg.log/var/log/dpkg.log.1
Rotated log archives:
/var/log/dpkg.log.*.gz
Files and folders access
Remote Access / Lateral movements
Name
Type
Description
Information / interpretation
Location
Tool(s)
Authorization (auth) logs
Remote Access / authentication information
Authentication information and sudo commands. More precisely, usage of authorization systems: successful or unsuccessful logins, sudo commands, etc.
Usually generated by the AUTH and AUTHPRIV facilities of the syslog daemon. AUTH regroups the authentication events / messages while AUTHPRIV regroups the elevation of privileges events / messages (such as commands executed through sudo).
Notably includes:
- Successful or unsuccessful logins to the sshd deamon. The authentication types (password, pubkey, etc.) or reason of failure (unknown user, invalid password) is specified.
- Commands executed with elevated privileges using sudo.
Location of the auth logs depend of the syslog daemon configuration (refer to the "Syslog daemon configuration" artefact below for more information).
[Debian / Ubuntu based systems]
Default location:
/var/log/auth.log/var/log/auth.log.1
Rotated log archives:
/var/log/auth.log.*.gz
[RedHat / CentOS based systems]
Default location (for AUTHPRIV logs):
/var/log/secure
Login records *tmp
Remote Access / authentication information
utmp / utmpx: currently logged users.
wtmp / wtmpx: all current and past logins, with additional details on system reboots, etc.
btmp / btmpx: all bad login attempts.
The *tmpx files are extended database files that supersede the *tmp files on some distributions.
*tmp login records are not stored in clear-text and must be parsed with adequate utilities, such as utmpdump <*TMP_FILE>.
SSH known hosts
Lateral movement
Possible SSH outgoing connections.
System-wide or user scoped known SSH keys for remote hosts. Usually collected, and user-validated, from the remote hosts when connecting for the first time.
The remote hosts hostname and IP address can be either stored in clear-text or hashed if HashKnownHosts is set to "yes" in the SSH client ssh_config configuration file.
Even if the hosts information are hashed, the following command can be used to check whether the specified hostname is present in the given known hosts file:
ssh-keygen -l -f <KNOWN_HOST_FILE> -F <HOSTNAME>.
Additionally, John can be used to bruteforce known hosts files:
john --format=known_hosts <KNOWN_HOST_FILE>nmap -sL -Pn -n 192.168.0.0/16 172.16.0.0/12 10.0.0.0/8 | grep '^Nmap scan report for' | cut -d ' ' -f 5 > IP_list.txtjohn --wordlist=IP_list.txt --format=known_hosts <KNOWN_HOST_FILE>
One of the few endpoint disk artefacts to identify outgoing SSH connections.
System-wide known hosts:
/etc/ssh/known_hosts<USER_HOME_DIR>/.ssh/known_hosts
Local persistence
Name
Type
Description
Information / interpretation
Location
Tool(s)
at jobs (atd daemon)
Persistence
Scheduled jobs that are configured using the at command-line utility to be run exactly one time. The jobs are executed as shell (bash, zsh, etc.) scripts.
By default, any user can create at jobs.
Each at jobs is represented by a file, which contains metadata information as comments, the environment variables for the execution, and the configured shell script / commands.
The filename follows a specific format ([a=]<JOB_NUMBER_5_CHAR><TIMESAMP_8_CHAR>) and gives additional information about the job. The following information can be deduced from the file name and the file itself:
- File created or last modified timestamp => when the at job was created.
- Username, uid, and gid of the user that created the job as shell comments in the file.
- Filename first char: a => job is pending or = => job is running.
- Filename 5 next chars => job id.
- Filename 8 next (and last) chars => hex-encoded minutes since epoch timestamp. Can be converted to retrieve the epoch timestamp of execution by converting to decimal and multiplying by 60.
Configured at jobs locations, each files representing a single at job:
/var/spool/at//var/spool/cron/atjobs/
Configuration files that define the users that can or cannot create at jobs:
/etc/at.allow/etc/at.deny
Output of currently running at jobs, saved as email text files:
/var/spool/at/spool//var/spool/cron/atspool/
Number of at jobs that have been created (already executed, executing, or scheduled):
/var/spool/at/.SEQ/var/spool/cron/atjobs/.SEQ
Trace of previous at jobs execution can be found in:
- Session opening by the atd daemon events in syslog or journal logs.
- at jobs email sent events in local email logs.
cron jobs (cron daemon)
Persistence
Scheduled jobs that, unlike at jobs, are executed repeatedly at a given frequency. cron jobs are executed at a pre-determined time (such as five minutes after midnight every day or at 2:15pm on the first of every month). The jobs are executed as shell (bash, zsh, etc.) scripts.
If the system is not running when a cron job is planned, the job will not be executed following the system boot.
By default, only root can define system-wide cron jobs (in /etc/crontab or under /etc/cron.d). But users can be allowed to use the crontab command, and will have their respective jobs created in the /var/spool/cron/crontabs/<USERNAME> file.
Usage rights of cron are defined by the /etc/cron.allow and /etc/cron.deny files:
- If cron.allow exists, only root and users listed in this file can use cron (and cron.deny is ignored).
- If only cron.deny exists, all users except the ones listed in this file can use cron.
- If neither of the files exist, only root can use cron (default state).
Each cron job is represented by a line in a crontab file.
The line format is a follow:
<MINUTE (0 - 59)> <HOUR (0 - 23)> <DAY (1 - 31)> <MONTH (1 - 12 / jan, feb, ...)> <DAY_WEEK (0 - 6 / mon, tue, ...)> <COMMAND>.
The specifiers can be replaced by a wildcard *, that represents "all". For example, a line with only wildcards (* * * * * <COMMAND>) means that the command will be executed every minute of every hour of every day of every month.
System-wide cron jobs:
/etc/crontab file.
crontab files under /etc/cron.d/.
User specific cron jobs, with one crontab file per user (that has configured jobs):
/var/spool/cron/crontabs/<USERNAME>.
Usage rights of cron:
/etc/cron.allow/etc/cron.deny
anacron jobs
Persistence
Scheduled jobs that are executed with a frequency specified in days or in a time interval (daily, weekly, monthly, or annually). Unlike cron jobs, anacron jobs are executed after a system boot if the system was not running during the last scheduled execution.
Depending on the Linux distribution, anacron itself may be run as a systemd timer (/lib/systemd/system/anacron.timer), a system cron job (/etc/cron.d/anacron), and / or at boot (/etc/init.d/anacron).
By default, cron "dot" directories (/etc/cron.{daily,weekly,monthly}) are often run by anacron.
Each anacron job is represented by a line in an anacrontab file.
The line format is a follow: <PERIOD_IN_DAYS> <DELAY_IN_MINUTES> <JOB_IDENTIFIER_STRING> <COMMAND>
Deprecated mechanism, in favor of Systemd / init.d, to define and start services as shell scripts at the system startup.
Scripts are configured to be executed at different run levels, from 0 (stop) to 6 (reboot) through 1 (maintenance mode) and 2-5 (multi-users mode, such as desktop startup, etc.).
System-wide or user scoped scripts that are executed by shells during their different stages of their initialization.
User scoped initialization script:
<USER_HOME_DIR>/.profile<USER_HOME_DIR>/.bash_profile<USER_HOME_DIR>/.zprofile<USER_HOME_DIR>/.bash_login<USER_HOME_DIR>/.zlogin
System-wide initialization scripts:
/etc/profile/etc/profile.d/*/etc/skel/.profile (Not used if <USER_HOME_DIR>/.bash_profile or <USER_HOME_DIR>/.bash_login exist).
Executed if an interactive shell is opened:
<USER_HOME_DIR>/.bashrc<USER_HOME_DIR>/.zshrc
Executed at the end of the session:
<USER_HOME_DIR>/.bash_logout/etc/zlogout<USER_HOME_DIR>/.zlogout
SSH authorization keys
Persistence
Specifies the SSH keys that can be used for logging into the user account for which the file is configured, thus allowing permanent access as that user.
Configuration of the SSH authorization keys:
/etc/ssh/sshd_configAuthorizedKeysFiledirective.
Default SSH authorization keys location:
<USER_HOME_DIR>/.ssh/authorized_keys<USER_HOME_DIR>/.ssh/authorized_keys2
XDG autostart entries
Persistence
XDG compliant graphical / desktop systems support XDG autostart entries, allowing applications to automatically start during the startup of a desktop environment (after user logon).
Each XDG autostart entry is represented by a file, which contains the following notable keys:
- Type key that specifies the entry type (application, link, or directory).
- Name key that indicates an arbitrary name assigned by the autostart entry creator.
- Exec key that defines the application and command line arguments to be executed.
User scoped initialization autostart entries:
<USER_HOME_DIR>/.config/autostart/*.desktop
System-wide autostart entries: /etc/xdg/autostart/*.desktop
Webshell
Command execution / Persistence.
Simply put, webshells are script files that are executed by a webserver. Webshells are notably leveraged to:
- execute code / commands on the underlying operating system following the exploitation of a web vulnerability (unrestricted file upload, remote code execution, etc.)
- maintain persistence following the compromise of an host exposing a webserver (usually Internet facing).
Webshells can be uncovered by:
- Yara rules aimed at webshells such as Neo23x0's gen_webshells.yar or thor-webshells.yar:
yara -r <YARA_RULE_PATH> <WEBSERVER_ROOT>
- Reviewing added files or modifications in legitimate files using code repository or a fresh install of the application if possible.
- Manually by looking for known webshell patterns (Runtime.getRuntime().exec, eval, system, etc.), obfuscated script files, or files modified during the targeted timeframe.
- Reviewing the webserver access logs if available, looking for exploitation IoCs, unusual requests, large response size, etc.
Web browsers and network usage
Name
Type
Description
Information / interpretation
Location
Tool(s)
wget HSTS history
Web browsers and network usage
wget utility's HTTP Strict Transport Security (HSTS) history.
HSTS is a mechanism to only allow access to a particular website in HTTPS if that website was accessed in HTTPS once and defines an HSTS policy. The HSTS policy to follow is define by the web server through the Strict-Transport-SecurityHTTP response header. The web browser or utility has to store the websites accessed in HTTPS (with HSTS implemented) for the duration specified in the header to support HSTS.
wget's HSTS history is implemented as a plaintext file, with an entry per line.
For each entry, the following notable information are available:
- Hostname of the accessed website
- Created timestamp in UTC (in epoch format) that defines when the entry was created. As the entry is overwritten upon new access to a website defining an HSTS policy, the created timestamp matches the last access to the website.
<USER_HOME_DIR>/.wget-hsts
-
Web servers and third-party applications logs
Name
Type
Description
Information / interpretation
Location
Tool(s)
Apache webserver logs
Web servers logs
Logs of the Apache webserver.
Debian / Ubuntu:
/var/log/apache2/access.log/var/log/apache2/error.log
RHEL / Red Hat / CentOS / Fedora :
/var/log/httpd/access_log/var/log/httpd/error_log
FreeBSD:
/var/log/httpd-access.log/var/log/httpd-error.log
Custom definition for access (CustomLog section) or error (ErrorLog section) logs:
/etc/httpd/conf/httpd.conf/etc/apache2/apache2.conf/usr/local/etc/apache22/httpd.conf
TODO
journactl
| systemd services | Persistence | The systemd service manager is used for managing background daemon processes, known as services.
systemd relies on service units | | System-wide service unit files:
/usr/lib/systemd/system/*/etc/systemd/system/*
User-scoped service unit files for user-level persistence:
<USER_HOME_DIR>/.config/systemd/user/ | yara rules |
Systemd utilizes configuration files known as service units to control how services boot and under what conditions. By default, these unit files are stored in the /etc/systemd/system and /usr/lib/systemd/system directories and have the file extension .service. Each service unit file may contain numerous directives that can execute system commands:
Systemd also defines several unit types:
Service: the unit starts, stops, restarts or reloads programs or daemons;
Socket: the unit is activated when it receives incoming traffic on a listening socket or in the context of inter-process communication;
Target: groups of units used at boot time to start the system in a particular state;
Device: the unit is activated when a device is connected to the Linux system;
Mount: the unit controls file system mount points;
Automount: the unit controls on-demand mounting of file system;
Timer: the unit is activated at specific times;
Swap: the unit encapsulates, activates or deactivates swap partitions or files;
Path: the unit is activated when a monitored file is modified;
Slice: group of units that manage system resources;
Scope: the unit organizes and manages foreign processes.
ExecStart, ExecStartPre, and ExecStartPost directives cover execution of commands when a services is started manually by 'systemctl' or on system start if the service is set to automatically start. ExecReload directive covers when a service restarts. ExecStop and ExecStopPost directives cover when a service is stopped or manually by 'systemctl'.
systemd provides its own network management features. It is based on three types of file:
.link files to configure physical network devices; .netdev files to configure virtual netowrk devices (VPN, tunnels...); .network files to configure the network layer. All these files are stored in the following directories:
/usr/lib/systemd/network/ (default files used by systemd daemons systemd-udevd and systemd-networkd); /etc/systemd/network/ (customized files by the system administrator).
Network Manager can also be used to manage network configuration /etc/NetworkManager.conf (general configuration information); /etc/NetworkManager/system-connections/ (configurations for each individual connection, declined by name).
DNS nameservers and hosts
/etc/resolv.conf /etc/resolv.conf.bak
The /etc/resolv.conf file is managed either by the openresolv resolvconf framwork or by the systemd's resolvconf framework. The systemd-resolved daemon is configured in the /etc/systemd/resolved.conf file. This file contains the daemon parameters, DNS servers, fallback servers and other DNS resolver configuration. The openresolv framwork stores the same information in the /etc/resolvconf.conf file.
/etc/hosts
/etc/network/if-up.d/upstart
/etc/apt/apt.conf.d - persistence
/etc/udev/rules.d/
<USER_HOME_DIR>/.viminfo
/etc/security/lastlog Specifies the path to the lastlog file.
/etc/group Contains the basic attributes of groups.
/etc/security/group Contains the extended attributes of groups.
/etc/passwd Contains the basic attributes of users.
/etc/security/passwd Contains password information. => Check user with shell and that have a different uid / gid than default.
/etc/security/environ Contains the environment attributes of users.
/etc/security/user Contains the extended attributes of users.
/etc/security/limits Contains the process resource limits of users. https://www.ibm.com/docs/en/aix/7.1?topic=formats-lastlog-file-format
USB devices activity will generate kernel logs, usually in /var/log/kern.log and Syslog centralized logs (such as /var/log/syslog) depending on the syslog daemon configuration.
