Artefacts overview

General

Name	Type	Description	Information / interpretation	Location	Tool(s)
Audit auditd framework (audit logs)	-	Non default, can be configured to log multiple types of operations, such as authentication successes or failures, process executions, file accesses, user commands executed in a TTY, etc.	Each record / log entry contain a msg field, composed of a timestamp and an unique ID. Multiple records generated as part of the same Auditd event can share the same msg field. For example, cat /etc/passwd can generate SYSCALL + EXECVE records for the execution of cat and a PATH record for the access to the /etc/passwd file. The type field contains the type of the record: - User authentication and access: USER_LOGIN_SUCCESS, USER_LOGIN_FAILED, USER_AUTH_SUCCESS, USER_AUTH_FAILED, USER_START_SUCCESS, USER_START_FAILED, SESSION_TERMINATED. - Process execution: EXECVE and SYSCALL. - Filesystem access: PATH (for relative or absolute file access), CWD (current working directory, useful to reconstruct full path if a relative path has been recorded in PATH records) and OPENAT. - Commands entered in a TTY console: TTY or by users: USER_CMD. - Full command-line of process: PROCTITLE. The associated proctitle field MAY be encoded in hexadecimal. - Network socket connections: SOCKADDR. The associated saddr field contains IP and port information, and can be interpreted directly at event generation (if log_format = ENRICHED is set), or with ausearch -i or simple scripting. - Account activity: ADD_USER or ADD_GROUP. - More record types are listed in the RedHat documentation. If present, the auid field defines the ID of the user upon login and remains the same even if the user's identity changes (for instance with su). If present, uid / gid and euid / egid fields define the user / group IDs and the effective user / group IDs of the audited process. If present, the tty and ses fields define respectively the terminal and session from which the audited process was invoked. For SYSCALL records, the aX field(s) define the arguments / parameters of the syscall, represented by unsigned long long integers and as such cannot be used to determine the values taken by the arguments.	Configuration file notably defining the path of the log files: /etc/auditd.conf Configuration defining the rules to apply: /etc/audit/audit.rules Rules best practice: https://github.com/Neo23x0/auditd Current log files (default location): /var/log/audit/audit.log /var/log/audit/audit.log.1 Rotated log archives (default location): /var/log/audit/audit.log.*.gz The aureport and ausearch utilities can (and if possible should) be used to search the auditd log files. Example: aureport -i [--login | --executable | ...] [--summary] -if <AUDIT_LOG_FILE>

System information

Name	Type	Description	Information / interpretation	Location	Tool(s)
alternatives logs	System information	Logs of the update-alternatives utility, used to manage alternatives (i.e symbolic links to a given command).		/var/log/alternatives.log
Environment variables information	System information.	Contains system-wide or user scoped persistent environment variables.	-	System-wide configuration file: /etc/environment Initialization scripts can also be used to define system-wide or user scoped environment variables.	-
Hostname information	System information	Contains the hostname of the system.	-	/etc/hostname	-
Mounted filesystems information	System information.	Contains information on the mounted file systems, such as partition types (ext3 / ext4, etc.).		Configuration: /etc/fstab Mount logs (such as Mounting operation / keyword): /var/log/dmesg	-
Timezone information	System information	Contains the timezone of the system.	-	/etc/timezone /etc/adjtime /etc/localtime	-
Syslog daemon configuration	System information	The Syslog deamon configuration file(s) notably define where the messages / events received by the Syslog daemon will be outputted. The messages are usually written as plaintext files under /var/log/ but can also be sent over the network.	Example of a configuration file writing logs to common files: auth,authpriv.* /var/log/auth.log *.*;auth,authpriv.none -/var/log/syslog kern.* -/var/log/kern.log mail.* -/var/log/mail.log	/etc/syslog.conf /etc/rsyslog.conf /etc/rsyslog.d/*.conf /etc/syslog­ng.conf /etc/syslog­ng/*	-

Filesystem

Overview

Contrary to NTFS partitions, file information is not stored in a specific location (such as the MFT) for ext* partitions but scattered across blocks or groups of blocks (contiguous blocks combined) across the partition. Blocks have a fixed size, determined at the time the filesystem is created.

On ext4, each block group notably store:

• The Super block, only replicated at the beginning of a fraction of block groups however, which contains various information about the file system: block size, location of the inode tables, size of block groups, etc.

• The inode bitmap and a data block bitmap that limit the number of inodes and data blocks of that particular block group.

• An inode table which is a linear array of inodes (first to last inode) of that particular block group.

  Inodes are data struct that define a file and each file is associated with one inode. An inode stores metadata about the file (size, owner UID / GID, permissions, timestamps, etc.) and (direct or indirect) pointers to data blocks that contain the file data, but does not store the file name and file data.

• Data blocks, that store file data.

Filesystem types supported timestamps

Filesystem	atime (access)	mtime (modification)	ctime (metadata change)	crtime (creation / birth)	Comment
ext2 ext3	x	x	x	-
ext4	x	x	x	x
XFS	x	x	x	x*	* Since XFS v5

Filesystem timelining

find <DIRECTORY> -xdev -print0 | xargs -0 stat -c 'crtime="%w" crtime_epoch="%W" mtime="%y" mtime_epoch="%Y" ctime="%z" ctime_epoch="%Z" atime="%x" atime_epoch="%X" size_bytes="%s" userID="%u" username="%U" groupID="%g" groupname="%G" access="%a" access_pretty="%A" filetype="%F" filename="%n" filename_deref="%N"'

Software installation and program execution

Name	Type	Description	Information / interpretation	Location	Tool(s)
apt / apt-get logs	Software installation	Logs of apt-get / apt operations, including packets installation.		Current log file: /var/log/apt/history.log Rotated log archives: /var/log/apt/history.log.*.gz
aptitude logs	Software installation.	Logs of the aptitude utility (front-end to apt) operations, including packets installation.		/var/log/aptitude
dpkg logs	Software installation.	Logs of dpkg operations, including packets installed / removed through the utility.		Current log files: /var/log/dpkg.log /var/log/dpkg.log.1 Rotated log archives: /var/log/dpkg.log.*.gz

Files and folders access

Remote Access / Lateral movements

Name	Type	Description	Information / interpretation	Location	Tool(s)
Authorization (auth) logs	Remote Access / authentication information	Authentication information and sudo commands. More precisely, usage of authorization systems: successful or unsuccessful logins, sudo commands, etc. Usually generated by the AUTH and AUTHPRIV facilities of the syslog daemon. AUTH regroups the authentication events / messages while AUTHPRIV regroups the elevation of privileges events / messages (such as commands executed through sudo).	Notably includes: - Successful or unsuccessful logins to the sshd deamon. The authentication types (password, pubkey, etc.) or reason of failure (unknown user, invalid password) is specified. - Commands executed with elevated privileges using sudo.	Location of the auth logs depend of the syslog daemon configuration (refer to the "Syslog daemon configuration" artefact below for more information). [Debian / Ubuntu based systems] Default location: /var/log/auth.log /var/log/auth.log.1 Rotated log archives: /var/log/auth.log.*.gz [RedHat / CentOS based systems] Default location (for AUTHPRIV logs): /var/log/secure
Login records *tmp	Remote Access / authentication information	utmp / utmpx: currently logged users. wtmp / wtmpx: all current and past logins, with additional details on system reboots, etc. btmp / btmpx: all bad login attempts. The *tmpx files are extended database files that supersede the *tmp files on some distributions.		Linux: /var/run/utmp /var/log/wtmp /var/log/btmp Solaris: /var/adm/utmp (deprecated) /var/adm/utmpx /var/adm/wtmp (deprecated) /var/adm/wtmpx FreeBSD 9.0: /var/run/utx.active (utmp equivalent) /var/log/utx.log (wtmp equivalent)	*tmp login records are not stored in clear-text and must be parsed with adequate utilities, such as utmpdump <*TMP_FILE>.
SSH known hosts	Lateral movement	Possible SSH outgoing connections. System-wide or user scoped known SSH keys for remote hosts. Usually collected, and user-validated, from the remote hosts when connecting for the first time.	The remote hosts hostname and IP address can be either stored in clear-text or hashed if HashKnownHosts is set to "yes" in the SSH client ssh_config configuration file. Even if the hosts information are hashed, the following command can be used to check whether the specified hostname is present in the given known hosts file: ssh-keygen -l -f <KNOWN_HOST_FILE> -F <HOSTNAME>. Additionally, John can be used to bruteforce known hosts files: john --format=known_hosts <KNOWN_HOST_FILE> nmap -sL -Pn -n 192.168.0.0/16 172.16.0.0/12 10.0.0.0/8 | grep '^Nmap scan report for' | cut -d ' ' -f 5 > IP_list.txt john --wordlist=IP_list.txt --format=known_hosts <KNOWN_HOST_FILE> One of the few endpoint disk artefacts to identify outgoing SSH connections.	System-wide known hosts: /etc/ssh/known_hosts <USER_HOME_DIR>/.ssh/known_hosts

Local persistence

Name	Type	Description	Information / interpretation	Location	Tool(s)
at jobs (atd daemon)	Persistence	Scheduled jobs that are configured using the at command-line utility to be run exactly one time. The jobs are executed as shell (bash, zsh, etc.) scripts. By default, any user can create at jobs.	Each at jobs is represented by a file, which contains metadata information as comments, the environment variables for the execution, and the configured shell script / commands. The filename follows a specific format ([a=]<JOB_NUMBER_5_CHAR><TIMESAMP_8_CHAR>) and gives additional information about the job. The following information can be deduced from the file name and the file itself: - File created or last modified timestamp => when the at job was created. - Username, uid, and gid of the user that created the job as shell comments in the file. - Filename first char: a => job is pending or = => job is running. - Filename 5 next chars => job id. - Filename 8 next (and last) chars => hex-encoded minutes since epoch timestamp. Can be converted to retrieve the epoch timestamp of execution by converting to decimal and multiplying by 60.	Configured at jobs locations, each files representing a single at job: /var/spool/at/ /var/spool/cron/atjobs/ Configuration files that define the users that can or cannot create at jobs: /etc/at.allow /etc/at.deny Output of currently running at jobs, saved as email text files: /var/spool/at/spool/ /var/spool/cron/atspool/ Number of at jobs that have been created (already executed, executing, or scheduled): /var/spool/at/.SEQ /var/spool/cron/atjobs/.SEQ Trace of previous at jobs execution can be found in: - Session opening by the atd daemon events in syslog or journal logs. - at jobs email sent events in local email logs.
cron jobs (cron daemon)	Persistence	Scheduled jobs that, unlike at jobs, are executed repeatedly at a given frequency. cron jobs are executed at a pre-determined time (such as five minutes after midnight every day or at 2:15pm on the first of every month). The jobs are executed as shell (bash, zsh, etc.) scripts. If the system is not running when a cron job is planned, the job will not be executed following the system boot. By default, only root can define system-wide cron jobs (in /etc/crontab or under /etc/cron.d). But users can be allowed to use the crontab command, and will have their respective jobs created in the /var/spool/cron/crontabs/<USERNAME> file. Usage rights of cron are defined by the /etc/cron.allow and /etc/cron.deny files: - If cron.allow exists, only root and users listed in this file can use cron (and cron.deny is ignored). - If only cron.deny exists, all users except the ones listed in this file can use cron. - If neither of the files exist, only root can use cron (default state).	Each cron job is represented by a line in a crontab file. The line format is a follow: <MINUTE (0 - 59)> <HOUR (0 - 23)> <DAY (1 - 31)> <MONTH (1 - 12 / jan, feb, ...)> <DAY_WEEK (0 - 6 / mon, tue, ...)> <COMMAND>. The specifiers can be replaced by a wildcard *, that represents "all". For example, a line with only wildcards (* * * * * <COMMAND>) means that the command will be executed every minute of every hour of every day of every month.	System-wide cron jobs: /etc/crontab file. crontab files under /etc/cron.d/. User specific cron jobs, with one crontab file per user (that has configured jobs): /var/spool/cron/crontabs/<USERNAME>. Usage rights of cron: /etc/cron.allow /etc/cron.deny
anacron jobs	Persistence	Scheduled jobs that are executed with a frequency specified in days or in a time interval (daily, weekly, monthly, or annually). Unlike cron jobs, anacron jobs are executed after a system boot if the system was not running during the last scheduled execution. Depending on the Linux distribution, anacron itself may be run as a systemd timer (/lib/systemd/system/anacron.timer), a system cron job (/etc/cron.d/anacron), and / or at boot (/etc/init.d/anacron). By default, cron "dot" directories (/etc/cron.{daily,weekly,monthly}) are often run by anacron.	Each anacron job is represented by a line in an anacrontab file. The line format is a follow: <PERIOD_IN_DAYS> <DELAY_IN_MINUTES> <JOB_IDENTIFIER_STRING> <COMMAND>	System-wide anacron jobs: /etc/anacrontab cron "dot" directories, that contain shell scripts directly: /etc/cron.daily/ /etc/cron.hourly/ /etc/cron.monthly/ /etc/cron.weekly/
Run Control (RC) scripts	Persistence	Deprecated mechanism, in favor of Systemd / init.d, to define and start services as shell scripts at the system startup. Scripts are configured to be executed at different run levels, from 0 (stop) to 6 (reboot) through 1 (maintenance mode) and 2-5 (multi-users mode, such as desktop startup, etc.).		RC scripts locations: /etc/rc.local /etc/rc.common /etc/rc<0-6>.d/* /etc/rcS.d/*
Shell initialization scripts	Persistence	System-wide or user scoped scripts that are executed by shells during their different stages of their initialization.		User scoped initialization script: <USER_HOME_DIR>/.profile <USER_HOME_DIR>/.bash_profile <USER_HOME_DIR>/.zprofile <USER_HOME_DIR>/.bash_login <USER_HOME_DIR>/.zlogin System-wide initialization scripts: /etc/profile /etc/profile.d/* /etc/skel/.profile (Not used if <USER_HOME_DIR>/.bash_profile or <USER_HOME_DIR>/.bash_login exist). Executed if an interactive shell is opened: <USER_HOME_DIR>/.bashrc <USER_HOME_DIR>/.zshrc Executed at the end of the session: <USER_HOME_DIR>/.bash_logout /etc/zlogout <USER_HOME_DIR>/.zlogout
SSH authorization keys	Persistence	Specifies the SSH keys that can be used for logging into the user account for which the file is configured, thus allowing permanent access as that user.		Configuration of the SSH authorization keys: /etc/ssh/sshd_config AuthorizedKeysFiledirective. Default SSH authorization keys location: <USER_HOME_DIR>/.ssh/authorized_keys <USER_HOME_DIR>/.ssh/authorized_keys2
XDG autostart entries	Persistence	XDG compliant graphical / desktop systems support XDG autostart entries, allowing applications to automatically start during the startup of a desktop environment (after user logon).	Each XDG autostart entry is represented by a file, which contains the following notable keys: - Type key that specifies the entry type (application, link, or directory). - Name key that indicates an arbitrary name assigned by the autostart entry creator. - Exec key that defines the application and command line arguments to be executed.	User scoped initialization autostart entries: <USER_HOME_DIR>/.config/autostart/*.desktop System-wide autostart entries: /etc/xdg/autostart/*.desktop
Webshell	Command execution / Persistence.	Simply put, webshells are script files that are executed by a webserver. Webshells are notably leveraged to: - execute code / commands on the underlying operating system following the exploitation of a web vulnerability (unrestricted file upload, remote code execution, etc.) - maintain persistence following the compromise of an host exposing a webserver (usually Internet facing).		Usual locations: /var/www/html /usr/local/www/ /etc/nginx /etc/apache2 /srv/ /srv/www ...	Webshells can be uncovered by: - Yara rules aimed at webshells such as Neo23x0's gen_webshells.yar or thor-webshells.yar: yara -r <YARA_RULE_PATH> <WEBSERVER_ROOT> - Reviewing added files or modifications in legitimate files using code repository or a fresh install of the application if possible. - Manually by looking for known webshell patterns (Runtime.getRuntime().exec, eval, system, etc.), obfuscated script files, or files modified during the targeted timeframe. - Reviewing the webserver access logs if available, looking for exploitation IoCs, unusual requests, large response size, etc.

Web browsers and network usage

Name	Type	Description	Information / interpretation	Location	Tool(s)
wget HSTS history	Web browsers and network usage	wget utility's HTTP Strict Transport Security (HSTS) history. HSTS is a mechanism to only allow access to a particular website in HTTPS if that website was accessed in HTTPS once and defines an HSTS policy. The HSTS policy to follow is define by the web server through the Strict-Transport-Security HTTP response header. The web browser or utility has to store the websites accessed in HTTPS (with HSTS implemented) for the duration specified in the header to support HSTS.	wget's HSTS history is implemented as a plaintext file, with an entry per line. For each entry, the following notable information are available: - Hostname of the accessed website - Created timestamp in UTC (in epoch format) that defines when the entry was created. As the entry is overwritten upon new access to a website defining an HSTS policy, the created timestamp matches the last access to the website.	<USER_HOME_DIR>/.wget-hsts	-

Web servers and third-party applications logs

Name	Type	Description	Information / interpretation	Location	Tool(s)
Apache webserver logs	Web servers logs	Logs of the Apache webserver.		Debian / Ubuntu: /var/log/apache2/access.log /var/log/apache2/error.log RHEL / Red Hat / CentOS / Fedora : /var/log/httpd/access_log /var/log/httpd/error_log FreeBSD: /var/log/httpd-access.log /var/log/httpd-error.log Custom definition for access (CustomLog section) or error (ErrorLog section) logs: /etc/httpd/conf/httpd.conf /etc/apache2/apache2.conf /usr/local/etc/apache22/httpd.conf

TODO

---

journactl

---

| systemd services | Persistence | The systemd service manager is used for managing background daemon processes, known as services. systemd relies on service units | | System-wide service unit files: /usr/lib/systemd/system/* /etc/systemd/system/* User-scoped service unit files for user-level persistence: <USER_HOME_DIR>/.config/systemd/user/ | yara rules |

Systemd utilizes configuration files known as service units to control how services boot and under what conditions. By default, these unit files are stored in the /etc/systemd/system and /usr/lib/systemd/system directories and have the file extension .service. Each service unit file may contain numerous directives that can execute system commands:

Systemd also defines several unit types:

• Service: the unit starts, stops, restarts or reloads programs or daemons;

• Socket: the unit is activated when it receives incoming traffic on a listening socket or in the context of inter-process communication;

• Target: groups of units used at boot time to start the system in a particular state;

• Device: the unit is activated when a device is connected to the Linux system;

• Mount: the unit controls file system mount points;

• Automount: the unit controls on-demand mounting of file system;

• Timer: the unit is activated at specific times;

• Swap: the unit encapsulates, activates or deactivates swap partitions or files;

• Path: the unit is activated when a monitored file is modified;

• Slice: group of units that manage system resources;

• Scope: the unit organizes and manages foreign processes.

ExecStart, ExecStartPre, and ExecStartPost directives cover execution of commands when a services is started manually by 'systemctl' or on system start if the service is set to automatically start. ExecReload directive covers when a service restarts. ExecStop and ExecStopPost directives cover when a service is stopped or manually by 'systemctl'.

[Unit] Description=Myservice [Service] ExecStart=/tmp/46868461631.tmp/malw4re

https://attack.mitre.org/techniques/T1543/002/

---

shell history

shell histories are only filled in when the shell is closed

---

init.d

---

Network configuration

Before systemd:

Red Hat and SUSE systems /etc/sysconfig/network-scripts/ /etc/sysconfig/network/ SUSE-only: /etc/wicked/

Debian-based systems

/etc/network/interfaces /etc/network/interfaces.d/

Systemd:

systemd provides its own network management features. It is based on three types of file:

.link files to configure physical network devices; .netdev files to configure virtual netowrk devices (VPN, tunnels...); .network files to configure the network layer. All these files are stored in the following directories:

/usr/lib/systemd/network/ (default files used by systemd daemons systemd-udevd and systemd-networkd); /etc/systemd/network/ (customized files by the system administrator).

Network Manager can also be used to manage network configuration /etc/NetworkManager.conf (general configuration information); /etc/NetworkManager/system-connections/ (configurations for each individual connection, declined by name).

---

DNS nameservers and hosts

/etc/resolv.conf /etc/resolv.conf.bak

The /etc/resolv.conf file is managed either by the openresolv resolvconf framwork or by the systemd's resolvconf framework. The systemd-resolved daemon is configured in the /etc/systemd/resolved.conf file. This file contains the daemon parameters, DNS servers, fallback servers and other DNS resolver configuration. The openresolv framwork stores the same information in the /etc/resolvconf.conf file.

/etc/hosts

---

/etc/network/if-up.d/upstart

/etc/apt/apt.conf.d - persistence

/etc/udev/rules.d/

<USER_HOME_DIR>/.viminfo

/etc/security/lastlog Specifies the path to the lastlog file.

/etc/group Contains the basic attributes of groups.

/etc/security/group Contains the extended attributes of groups.

/etc/passwd Contains the basic attributes of users.

/etc/security/passwd Contains password information. => Check user with shell and that have a different uid / gid than default.

/etc/security/environ Contains the environment attributes of users.

/etc/security/user Contains the extended attributes of users.

/etc/security/limits Contains the process resource limits of users. https://www.ibm.com/docs/en/aix/7.1?topic=formats-lastlog-file-format

USB devices activity will generate kernel logs, usually in /var/log/kern.log and Syslog centralized logs (such as /var/log/syslog) depending on the syslog daemon configuration.

---

motd message of the day

https://pberba.github.io/security/2022/02/06/linux-threat-hunting-for-persistence-initialization-scripts-and-shell-configuration/#10-boot-or-logon-initialization-scripts-motd

---

---

References

https://www.sciencedirect.com/science/article/pii/S1742287612000357

https://wiki.debian-fr.xyz/Consulter_les_logs_:_quoi,o%C3%B9_et_comment_chercher%3F

https://nostarch.com/download/samples/PracticalLinuxForensics_Ch5_072721.pdf

https://blog.codeasite.com/how-do-i-find-apache-http-server-log-files/

https://sematext.com/blog/auditd-logs-auditbeat-elasticsearch-logsene/

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/security_guide/sec-understanding_audit_log_files

https://www.elastic.co/fr/blog/grokking-the-linux-authorization-logs

https://unix.stackexchange.com/questions/31549/is-it-possible-to-find-out-the-hosts-in-the-known-hosts-file

https://pberba.github.io/security/2021/11/22/linux-threat-hunting-for-persistence-sysmon-auditd-webshell/

https://en.wikipedia.org/wiki/Utmp

https://pberba.github.io/security/2022/02/06/linux-threat-hunting-for-persistence-initialization-scripts-and-shell-configuration/

https://attack.mitre.org/techniques/T1547/013/

https://unix.stackexchange.com/questions/411051/how-does-anacron-work-if-its-not-a-daemon

https://linux.die.net/man/8/cron

https://linux.die.net/man/5/crontab

https://linux.die.net/man/8/anacron

https://linux.die.net/man/5/anacrontab