Live Forensic on Linux

Last month, I wrote a bit about doing live forensic on a Windows machine. Today, let's do Linux.

Let's do a bit of recall before we proceed. Since I'm lazy to repeat, here are excerpts of what I have written previously in Live Forensic on Windows:

Before we touch that, why do we need to do live forensic at the first place? For a few reasons:
a) It is a production server and the Business Owner or System Admin would not let you shut down the system/server for offline forensic
b) The server/system is at a location that you could not go there physically
c) We afraid that we may lost crucial information e.g. malware that runs in memory only if we were to shut down the system immediately

Next, what info or data should we gather? What tools to use? In IT Forensic, we normally talk about using trusted binaries. Why is it important? Because on a hacked or malware infected machines, it is not uncommon for the attacker/malware to install rootkits or replace some common commands/binaries of the system/server in order to hide or cover their tracks. Running these binaries might not give you the real output or info as they should be. Therefore, the first steps is to prepare a forensic kit (e.g. write protected USB stick, CD) with your trusted binaries/tools.

Now, what tools you can use? Unlike Windows, Linux binaries are quite sensitive to the kernel's version. Also, have you heard about dynamic library dependency hell? Basically one library depends on other library which depends on another libraries and so on... Thus, most of the time you can't just copy out the binary/program and expect it to work on another system. You can always compile your own binary statically, but that require lots of works as well. Luckily, I found a saviour - Busybox! Yes, it is the same tool you use to run commands on your rooted Android devices :)

So, go grab yourself the Linux version of Busybox now!

For memory dump acquisition:
1. Use LiME. However, it might not work if the system prevent loading of kernel module. it is also very kernel specific, thus you can't compile it on a system and expect it to work on any systems. It will only work on a system with a same kernel version. 
2.  dd if=/dev/mem of=host1/dd-dev-mem.img . However, this may not work with newer kernel or if the kernel is compiled with STRICT_DEVMEM=y option (check /boot/config-<KERNELVERSION>). 

Have fun!

No.	What to Acquire	Tools/Commands to Use (Output is saved to a file)
1.	Hostname	·         ./busybox-i686 hostname > targethost/b-hostname.txt ·         hostname > targethost/hostname.txt
2.	OS version	·         ./busybox-i686 uname –a > targethost/b-uname-a.txt ·         uname –a > targethost/uname-a.txt ·         cat /etc/os-release > targethost/os-release.txt
3.	Current system date and time	·         ./busybox-i686 date > targethost/b-date.txt ·         date > targethost/date.txt
4.	Current IP address	·         ./busybox-i686 ifconfig > targethost/b-ifconfig.txt ·         ifconfig –a > targethost/ifconfig-a.txt
5.	Current running process list	·         ./busybox-i686 ps –eaf > targethost/b-ps-eaf.txt ·         ps –eaf > targethost/ps-eaf.txt ·         ./busybox-i686 lsof  –a > targethost/b-lsof.txt ·         lsof > targethost/lsof.txt
6.	current network connection lis	·         ./busybox-i686 netstat –anp > targethost/b-netstat-anp.txt ·         netstat –anp > targethost/netstat-anp.txt ·         ./busybox-i686 netstat –anr > targethost/b-netstat-anr.txt ·         netstat –anr > targethost/netstat-anr.txt
7.	current list of current logon sessions	·         ./busybox-i686 who –a > targethost/b-who-a.txt ·         who –a > targethost/who-a.txt ·         w > targethost/w.txt
8.	list of auto start applications and services	·         chkconfig --list > targethost/chkconfig--list.txt ·         ./busybox-i686 ls –alR /etc/rc* > targethost/ls-al-etc-rc.txt ·         ./busybox-i686 ls –alR /etc/init.d > targethost/ls-al-rc-d.txt ·         more /etc/init.d/* > targethost/more-init-d.txt ·         cat /etc/inittab > targethost/inittab.txt ·         service –-status-all > targethost/service—status-all.txt ·         ./busybox-i686 ls -alR /etc/systemd* > targethost/ls-al-etc-systemd.txt ·         ./busybox-i686 cat /etc/inetd.conf > targethost/inetd.conf ·         cat /etc/inetd.conf > targethost/inetd.conf
9.	environment variables	·         ./busybox-i686 env > targethost/b-env.txt ·         env > targethost/env.txt
10.	list of cron jobs (scheduler)	·         ./busybox-i686 cat /etc/crontab > targethost/b-crontab.txt ·         cat /etc/crontab > targethost/crontab.txt
11.	system event (dmesg) log records	·         ./busybox-i686 dmesg > targethost/b-dmesg.txt ·         dmesg > targethost/dmesg.txt
12.	last user activity records	·         ./busybox-i686 last > targethost/b-last.txt ·         last > targethost/last.txt ·         lastb > targethost/lastb.txt ·         lastlog > targethost/lastlog.txt
13.	list of installed software	·         rpm –qa targethost/rpm-qa.txt ·         dpkg --get-selections > targethost/dpkg—get-selections.txt
14.	list of user accounts	·         ./busybox-i686 cat /etc/passwd > targethost/b-passwd.txt ·         cat /etc/passwd > targethost/passwd.txt ·         ./busybox-i686 cat /etc/group > targethost/b-group.txt ·         cat /etc/group > targethost/group.txt
15.	partition table and drive info	·         ./busybox-i686 df –h > targethost/b-df-h.txt ·         df –h > targethost/df-h.txt ·         ./busybox-i686 fdisk -l > targethost/b-fdisk-l.txt ·         fdisk -l > targethost/fdisk-l.txt ·         parted –l targethost/parted-l.txt ·         ./busybox-i686 cat /etc/fstab > targethost/b-fstab.txt ·         cat /etc/fstab > targethost/fstab.txt ·         ./busybox-i686 mount > targethost/b-mount.txt ·         mount > targethost/mount.txt
16.	list of loaded modules	·         ./busybox-i686 lsmod > targethost/b-lsmod.txt ·         lsmod > targethost/lsmod.txt ·         ./busybox-i686 cat /proc/modules > targethost/b-proc-modues.txt ·         cat /proc/modules > targethost/proc-modues.txt
17.	information about memory usage	·         ./busybox-i686 cat /proc/meminfo > targethost/b-proc-meminfo.txt ·         cat /proc/meminfo > targethost/proc-meminfo.txt
18.	iptables rules (firewall)	·         iptables --list > targethost/iptables--list.txt
19.	system logs normally stored in /var/log	·         ./busybox-i686 tar –czvf targethost/b-var-log.tgz /var/log
20.	memory dump with LiME	As the LiME software needs to be specially built for the target system Linux’s kernel, there are more steps to be done before the tool can be used: a.     Extract the LiME source file you downloaded. b.     Change directory into the “src” directory. Type: cd src c.     Compile the module. Type: make ·         If successful, a new file starting with “lime’ and ending with “.ko” will be created. Example: lime-3.2.6.ko ·         insmod lime*.ko “path=targethost/lime.mem format=lime” The module is then loaded to the kernel and the memory dump will happen automatically. If you need to run it again, you must first remove the module from the kernel. Type: rmmod lime
21.	/dev/mem and /dev/kmem via dd	·         dd if=/dev/mem of=targethost/dd-dev-mem.img ·         dd if=/dev/kmem of=targethost/dd-dev-kmem.img

Live forensic on Windows

In the last posts, I talked about the processes of IT Forensic. Those are just theories as one might say. Hence, today, let's get more real and technical. How about what can we do to perform live forensic on Windows systems sounds to you?

Before we touch that, why do we need to do live forensic at the first place? For a few reasons:
a) It is a production server and the Business Owner or System Admin would not let you shut down the system/server for offline forensic
b) The server/system is at a location that you could not go there physically
c) We afraid that we may lost crucial information e.g. malware that runs in memory only if we were to shut down the system immediately

Next, what info or data should we gather? What tools to use? In IT Forensic, we normally talk about using trusted binaries. Why is it important? Because on a hacked or malware infected machines, it is not uncommon for the attacker/malware to install rootkits or replace some common commands/binaries of the system/server in order to hide or cover their tracks. Running these binaries might not give you the real output or info as they should be. Therefore, the first steps is to prepare a forensic kit (e.g. write protected USB stick, CD) with your trusted binaries/tools.

Back to what info or data to gather.... Below are the lists of what we should gather and how or using what tool can we gather it (yes, I know some of them are not of trusted binaries but there are also advantages in running them. Can you see those advantages? Answers below :) ) . Basically, we will need these free and great software suites:
a) Sysinternals Suite - http://technet.microsoft.com/en-us/sysinternals/
b) Nirsoft - http://www.nirsoft.net/
c) Redline (for memory dump and analysis) - http://www.mandiant.com/resources/download/redline

No.	What to Acquire	Tools/Commands to Use (Output is saved to a file)
1.	Hostname, OS version, system info, list of software installed	·         Psinfo –h –s –d > targethost\psinfo-hsd.txt
2.	System info	·         Systeminfo > targethost\systeminfo.txt
3.	Current system date	·         Date /t > targethost\date.txt
4.	Current system time	·         Time /t > targethost\time.txt
5.	Registry dump	·         Regedit /E targethost\registry.txt
6.	Current IP address	·         Ipconfig /all > targethost\ipconfig.txt ·         Awatch /stab targethost\awatch.txt ·         Networkinterfacesview targethost\networkinterfacesview.csv
7.	Running current process list	·         Pslist –t > targethost\pslist-d.txt ·         Pslist –x > targethost\pslist-x.txt
8.	Current network connection list	·         Netstat –anb > targethost\netstat-anb.txt ·         Netstat –anr > targethost\netstat-anr.txt ·         Tcpvcon –an > targethost/tcpvcon-an.txt ·         Cports.exe /scomma targethost\cports.csv
9.	Current list of current logon sessions	·         Psloggedon > targethost\psloggedon.txt
10.	List of auto start applications	·         Autorunsc –a –c > targethost\autorunsc.csv
11.	Environment variables	·         Set > targethost\set.txt
12.	List of services	·         Psservice > targethost\psservice.txt
13.	System event log records	·         Psloglist –x > targethost\psloglist-x.txt
14.	Last user activity records	·         Lastactivityview /scomma  targethost\lastactivityview.csv
15.	Windows turn on/off time records	·         Turnontimesview /scomma  targethost\turnontimesview.csv
16.	Windows user login/logoff records	·         Winlogonview /scomma  targethost\winlogonview.csv
17.	List of installed software	·         Myuninst /stab targethost\myuninst.csv
18.	List of loaded dlls	·         Listdlls > targethost\listdlls.txt
19.	List of user accounts	·         Net user > targethost\net-user.txt ·         Userprofilesview /scomma  targethost\userprofilesview.csv
20.	Browser history ·         IE ·         Mozilla ·         Chrome ·         Opera ·         Safari	·         Iecacheview /stab  targethost\iecacheview.csv ·         Iehv /stab targethost\iehv.csv ·         Mozillacacheview /scomma targethost\mozillacacheview.csv ·         Mozillahistoryview /scomma targethost\mozillahistoryview.csv ·         Mzcv /stab targethost\mzcv.csv ·         Chromecacheview /scomma targethost\chromecacheview.csv ·         Chromehistory /scomma targethost\chromehistoryview.csv ·         Chromecookiesview /scomma targethost\chromecookiesview.csv ·         Operacacheview /scomma targethost\operacacheview.csv ·         Safarihistoryview /scomma targethost\safarihistoryview.csv ·         Safaricacheview /scomma targethost\safaricacheview.csv
21.	Using Redline Comprehensive Collector tool to acquire full memory dump, page file data, running processes, registry data etc.	·         Runredlineaudit.bat

Have fun!

Oh wait... what are the advantages of running (carefully) "untrusted" binaries as well?
By comparing the output of trusted binaries vs untrusted binaries, it may give us some clues that the "untrusted binaries" had been modified or "rookitted". We then can analyse this binary in order to track down other malicious binaries  that might get installed on the system as well.

Enterprise IT Forensic Process - Disposal

In my previous posts, I have covered the first 4 processes of Enterprise IT Forensic Process:

1) Approval - Ensuring that we are allowed to do what we want to do
2) Acquisition - Ensuring that we collect and acquire the evidence in a forensically sound manner
3) Analysis - Performing the analysis and investigation, also in a forensically sound manner
4) Reporting - What a report should contain?

Disposal process is the final piece of the puzzle.

Once we have done the analysis, completed the report, the next question is what to do with the evidence (both original and acquired) that we have gathered? We cannot keep the evidence forever due to various reasons, e.g. storage limitation, legal requirements, security etc.

Basically, the options are:
1) Store - If there is a need to preserve the evidence e.g. legal case
2) Return - Return the evidence to the owner or data custodian
3) Forward - Forward the evidence to another party as agreed with the Requestor
4) Dispose - Securely delete or dispose the evidence

However, it is important to take note that the above decision does not lies entirely with the forensic examiner or investigator. The decision shall be made together with the Requestor.

There is also possibilities that the Requestor might want the original evidence to be returned and the acquired evidence to be deleted or vise versa. Anyway, regardless of the option, the chain of custody must be maintained and updated to reflect the status.

To delete/wipe an evidence, for example a hard drive, simply formatting the hard drive is not secure enough as data can still be recovered. There are a few methods out there that you could use to securely wipe a drive:

1) Hardware based - It is the fastest way. The hardware is known as degausser. It will render the drive useless digitally and physically - not a good idea if you still want to use the drive for other purposes.
2) Software based  - This method which is slower is to rewrite the drive with zeros or random data multiple times. There are various free tools out there that can do the job. For example diskwipe and dban.

Enterprise IT Forensic Process - Reporting

So far, I have covered 3 of the 5 key processes of Enterprise IT Forensic Process:

1) Approval - Ensuring that we are allowed to do what we want to do
2) Acquisition - Ensuring that we collect and acquire the evidence in a forensically sound manner
3) Analysis - Performing the analysis and investigation, also in a forensically sound manner

The next one is Reporting. Well, I agree, there is no rocket science about this one.

First of all, a forensic report should be written purely based on evidence and reference to the mentioned evidence must be made clear. An examiner or investigator should not write something based on assumptions.

In general, a forensic report should contain the following:
1) Introduction - Describe the background of the forensic investigation.
2) Objective - Describe the objective of the investigation. What is the purpose, what you were asked to look  for.
3) Executive Summary -  This section is to provide quick management summary. State the main highlights or findings and summary or conclusion.
4) Detailed Observations - List all observations, in detail , with reference to evidence. For example:

Based on the email (REF: Appendix 1, Item No.8) sent out by the suspect (Joe Black) to Mary Margaret on 18 Jul 2012 04:21:02 AM, it is possible that the suspect was aware that the invoice (REF: Appendix 1, Item No.10) that were given to her (Jane Doe) was a forged one.

5) Evidence Information - Detailed information of all evidence obtained and analysed.
6) Appendix - List all referred evidence and their contents here.

Enterprise IT Forensic Process - Analysis

In the last two months, I have talked about the first two processes - Approval and Acquisition. Now, let's us move to the next process - Analysis. 

You may have heard of PPT - People, Process and Technology.  While the Approval and Acquisition are more about Process and Technology, Analysis is really about People. No matters how good your processes or technologies are, without the "People" factor, those processes or technologies would not yield much tangible outcome. One needs to have a very good analytical skills and adequate experience to be a good forensic examiner. One gains experience by doing more forensics in different scenarios and solving more technical issues etc. Bottom line, it's all about experience. 

Nevertheless, there is one vehemently crucial element for the Analysis process. Even the most experience forensic examiner will need to have this prior to any investigation:

Knowing what to look for - You can't find anything if you don't know what to look for.  For example, one cannot just tell the police to look for a "murderer" in a big shopping complex. The police would need more detail descriptions of the murderer - male or female? Hair colour? What type of clothes etc.  It is the same in IT forensic, one cannot just throw a laptop to a forensic examiner and tell him/her to look for something criminal on this laptop. It needs to be more specific than that. For example - "look for any trace of child pornography in this laptop" is specific. 

This info about "What to look for" shall be obtained prior to Approval process, ideally it should be part of the Request for Investigation. 

Once you know what to look for, the next steps will be:

How to look for - There is no fix procedure or formula for this. It's really depends on situation and it is case by case. This is when one's experience really make a hell lots of difference. However, as a start, in most cases a forensic investigator or an examiner can use a certain forensic tool such as Encase or FTK to do a search based on relevant keywords. The search results would give more hints or clues on what or where to look deeper. In a nutshell, here are the basic steps:

1. Develop basic keywords
2. Perform search based on those keywords
3. Review search results
4. Refine keywords or develop new keywords
5. Repeat 2 - 4 until tangible results are obtained. 
6. Mark, note or extract those relevant evidence for reporting later. 

Of course, the above approach may not be always valid or applicable. For example, if you are investigating a DoS attack, you'll need to use a completely different approach. Using Encase or FTK to review firewall, routers, webservers logs are not effective and I will say it doesn't even make sense to do so. For this one, manual reviews of the logs with some customised filtering scripts is the best way forward. Needless to say, every investigator has his/her own favourite tools and methods.

Enterprise IT Forensic Process - Acquisition

Last month, I talked about the first  process in Enterprise IT Forensic Process, which is the Approval process. Today, I shall proceed to talk about the next process - Acquisition.

What is acquisition? In a nutshell, it means collecting the evidence. Sounds easy right? Not really. There are many things need to be considered, especially if there is a high chance that the investigation will lead into a legal case.

You may have heard that the evidence collection must be done in a forensically sound manner. I bet you would recall from many scenes in CSI or other movies alike that some criminals got away scott-free on technical grounds, for example police's mistake when taking evidence etc. This sort of things could happen in IT Forensic as well. Thus, it is very imperative that the acquisition is done properly and (again), in a forensically sound manner.

Now, what is meant by "in a forensically sound manner"? Basically:
1. Ensure that evidence intake is done legally (refer my last piece on "Approval" process).
2. Evidence's chain of custody is well documented and preserved.
3. Ensuring that tampering of evidence is not possible during collecting, transferring, analysis and storing of the evidence.
4. All forensic activities are well documented and traceable.

I will say a) Evidence Intake and b) Evidence Chain of Custody are two key sub-processes within the Acquisition Process. Furthermore, there are two principals that I always apply:
1) Four Eyes Principal - ensuring that there is always a witness around
2) Bag and Tag - ensuring that evidence is properly labelled, sealed and its movements are recorded.

a) Evidence Intake:
Basically referring to how the evidence is collected or taken into custody.

Let's image a simple and basic scenario - A forensic investigator is tasked to collect a laptop from IT department (data custodian) for forensic.

What the forensic investigator needs to prepare beforehand?

Tools:
1. A camera or phone with decent camera - It is always a good idea to photograph everything before you touch the evidence.
2. Evidence Intake and Custody form:
a) To record the information of the to-be-taken evidence
b) This form also serves as an acknowledgement of transfer/receipt of evidence. Both the investigator and the custodian shall sign on it (Four Eyes Principal).
3. Waterproof envelope to "bag" the evidence
4. Sticker to "tag" (labelling) the evidence

General Steps:
1. First, take photos of the evidence. Important info such as serial number, model, brand etc shall be clearly photographed.
2. Fill in the form and record all details as possible e.g. the model of the laptop, serial no, HDD size, its condition etc.
3. Label the evidence (Tag) with a unique ID (you shall already has this info beforehand!).
4. Put the evidence into the envelop and seal it (Bag).
5. Label the sealed envelope (Tag).
6. Sign the form and ensure that the data custodian counter signs as well (Four Eyes Principal). Once both parties have signed, the custody of the evidence is now with the investigator.
7. The investigator can now proceed to his lab and start the forensic analysis (shall be done in a forensically sound manner as well).

That's it for a simple scenario. How about a more complex scenario? Such as - evidence intake is to-be-done by a representative in a remote location, then the evidence will be shipped to the head office and handed over personally to investigator. As you can imagine, the evidence intake and chain of custody process will be much more complex.  So, stay tuned for the next piece!