devops-exercises
Linux
Linux Master Application
A completely free application for testing your knowledge on Linux. Disclaimer: developed by repository owner
- Linux
- Linux Master Application
- Linux Exercises
- Linux Questions
- Linux 101
- I/O Redirection
- Filesystem Hierarchy Standard
- Permissions
- Scenarios
- Systemd
- Troubleshooting and Debugging
- Kernel
- SSH
- Globbing & Wildcards
- Boot Process
- Disk and Filesystem
- Performance Analysis
- Processes
- Security
- Networking
- DNS
- Packaging
- DNF
- Applications and Services
- Users and Groups
- Hardware
- Namespaces
- Virtualization
- AWK
- System Calls
- Filesystem & Files
- Advanced Networking
- Memory
- Distributions
- Sed
- Misc
Linux Exercises
Basics
| Name | Topic | Objective & Instructions | Solution | Comments |
|---|---|---|---|---|
| Navigation | cd, pwd | Exercise | Solution | |
| Create and Destroy | touch, rm, mkdir | Exercise | Solution | |
| Copy Time | touch, cp, ls | Exercise | Solution |
Misc
| Name | Topic | Objective & Instructions | Solution | Comments |
|---|---|---|---|---|
| Unique Count | Exercise | Solution |
Linux Questions
Linux 101
What is Linux?
[Wikipedia](https://en.wikipedia.org/wiki/Linux): "Linux is a family of open-source Unix-like operating systems based on the Linux kernel, an operating system kernel first released on September 17, 1991, by Linus Torvalds. Linux is typically packaged in a Linux distribution." [Red Hat](https://www.redhat.com/en/topics/linux/what-is-linux): "Linux® is an open source operating system (OS). An operating system is the software that directly manages a system’s hardware and resources, like CPU, memory, and storage. The OS sits between applications and hardware and makes the connections between all of your software and the physical resources that do the work."
Explain what each of the following commands does and give an example on how to use it: * touch * ls * rm * cat * cp * mkdir * pwd * cd
* touch - update file's timestamp. More commonly used for creating files * ls - listing files and directories * rm - remove files and directories * cat - create, view and concatenate files * cp - copy files and directories * mkdir - create directories * pwd - print current working directory (= at what path the user currently located) * cd - change directory
What each of the following commands does? * cd / * cd ~ * cd * cd .. * cd . * cd -
* cd / -> change to the root directory * cd ~ -> change to your home directory * cd -> change to your home directory * cd .. -> change to the directory above your current i.e parent directory * cd . -> change to the directory you currently in * cd - -> change to the last visited path
Some of the commands in the previous question can be run with the -r/-R flag. What does it do? Give an example to when you would use it
The -r (or -R in some commands) flag allows the user to run a certain command recursively. For example, listing all the files under the following tree is possible when done recursively (`ls -R`): /dir1/ dir2/ file1 file2 dir3/ file3 To list all the files, one can run `ls -R /dir1`
Explain each field in the output of `ls -l` command
It shows a detailed list of files in a long format. From the left: * file permissions, number of links, owner name, owner group, file size, timestamp of last modification and directory/file name
What are hidden files/directories? How to list them?
These are files directly not displayed after performing a standard ls direct listing. An example of these files are .bashrc which are used to execute some scripts. Some also store configuration about services on your host like .KUBECONFIG. The command used to list them is, `ls -a`
What do > and < do in terms of input and output for programs?
They take in input (<) and output for a given file (>) using stdin and stdout. `myProgram < input.txt > executionOutput.txt`
Explain what each of the following commands does and give an example on how to use it: * sed * grep * cut * awk
- sed: a stream editor. Can be used for various purposes like replacing a word in a file: `sed -i s/salad/burger/g` - grep: a search tool. Used to search, count or match a text in a file: - searching for any line that contains a word in a file: `grep 'word' file.md` - or displaying the total number of times a string appears in a file: `grep -c 'This is a string' file.md` - cut: a tool for cutting out selected portions of each line of a file: - syntax: `cut OPTION [FILE]` - cutting first two bytes from a word in a file: `cut -b 1-2 file.md`, output: `wo` - awk: a programming language that is mainly used for text processing and data extraction. It can be used to manipulate and modify text in a file: - syntax: awk [OPTIONS] [FILTER] [FILE] extracting a specific field from a CSV file: awk -F ',' '{print $1}' file.csv, output: first field of each line in the file
How to rename the name of a file or a directory?
Using the `mv` command.
Specify which command would you use (and how) for each of the following scenarios * Remove a directory with files * Display the content of a file * Provides access to the file /tmp/x for everyone * Change working directory to user home directory * Replace every occurrence of the word "good" with "great" in the file /tmp/y
- `rm -rf dir` - `cat or less` - `chmod 777 /tmp/x` - `cd ~` - `sed -i s/good/great/g /tmp/y`
How can you check what is the path of a certain command?
* whereis * which
What is the difference between these two commands? Will it result in the same output? ``` echo hello world echo "hello world" ```
The echo command receives two separate arguments in the first execution and in the second execution it gets one argument which is the string "hello world". The output will be the same.
Explain piping. How do you perform piping?
Using a pipe in Linux, allows you to send the output of one command to the input of another command. For example: `cat /etc/services | wc -l`
Fix the following commands: * sed "s/1/2/g' /tmp/myFile * find . -iname \*.yaml -exec sed -i "s/1/2/g" {} ;
``` sed 's/1/2/g' /tmp/myFile # sed "s/1/2/g" is also fine find . -iname "*.yaml" -exec sed -i "s/1/2/g" {} \; ```
How to check which commands you executed in the past?
history command or .bash_history file * also can use up arrow key to access or to show the recent commands you type
Running the command df you get "command not found". What could be wrong and how to fix it?
Most likely the default/generated $PATH was somehow modified or overridden thus not containing /bin/ where df would normally go.
This issue could also happen if bash_profile or any configuration file of your interpreter was wrongly modified, causing erratics behaviours.
You would solve this by fixing your $PATH variable:
As to fix it there are several options:
1. Manually adding what you need to your $PATH PATH="$PATH":/user/bin:/..etc
2. You have your weird env variables backed up.
3. You would look for your distro default $PATH variable, copy paste using method #1
Note: There are many ways of getting errors like this: if bash_profile or any configuration file of your interpreter was wrongly modified; causing erratics behaviours,
permissions issues, bad compiled software (if you compiled it by yourself)... there is no answer that will be true 100% of the time.
How do you schedule tasks periodically?
You can use the commands
cron and at.
With cron, tasks are scheduled using the following format:
*/30 * * * * bash myscript.sh Executes the script every 30 minutes.
crontab -e
Alternatively if you are using a distro with systemd it's recommended to use systemd timers.
</b></details>
### I/O Redirection
Explain Linux I/O redirection
In Linux, IO redirection is a way of changing the default input/output behavior of a command or program. It allows you to redirect input and output from/to different sources/destinations, such as files, devices, and other commands. Here are some common examples of IO redirection: * Redirecting Standard Output (stdout):
ls > filelist.txt
* Redirecting Standard Error (stderr):
ls /some/nonexistent/directory 2> error.txt
* Appending to a file:
echo "hello" >> myfile.txt
* Redirecting Input (stdin):
sort < unsorted.txt
* Using Pipes: Pipes ("|"):
ls | grep "\.txt$"
Demonstrate Linux output redirection
ls > ls_output.txt
Demonstrate Linux stderr output redirection
yippiekaiyay 2> ls_output.txt
Demonstrate Linux stderr to stdout redirection
yippiekaiyay &> file
What is the result of running the following command? yippiekaiyay 1>&2 die_hard</code>
An output similar to: `yippikaiyay: command not found...`
The file `die_hard` will not be created
In Linux FHS (Filesystem Hierarchy Standard) what is the /?
The root of the filesystem. The beginning of the tree.
What is stored in each of the following paths? - /bin, /sbin, /usr/bin and /usr/sbin - /etc - /home - /var - /tmp
* binaries * configuration files * home directories of the different users * files that tend to change and be modified like logs * temporary files
What is special about the /tmp directory when compared to other directories?
`/tmp` folder is cleaned automatically, usually upon reboot.
What kind of information one can find in /proc?
It contains useful information about the processes that are currently running, it is regarded as control and information center for kernel.
What makes /proc different from other filesystems?
True or False? only root can create files in /proc
False. No one can create file in /proc directly (certain operations can lead to files being created in /proc by the kernel).
What can be found in /proc/cmdline?
The command passed to the boot loader to run the kernel
In which path can you find the system devices (e.g. block storage)?
/dev
How to change the permissions of a file?
Using the `chmod` command.
What does the following permissions mean?: * 777 * 644 * 750
777 - You give the owner, group and other: Execute (1), Write (2) and Read (4); 4+2+1 = 7. 644 - Owner has Read (4), Write (2), 4+2 = 6; Group and Other have Read (4). 750 - Owner has x+r+w, Group has Read (4) and Execute (1); 4+1 = 5. Other have no permissions.
What this command does? chmod +x some_file
It adds execute permissions to all sets i.e user, group and others
Explain what is setgid and setuid
* setuid is a linux file permission that permits a user to run a file or program with the permissions of the owner of that file. This is possible by elevation of current user privileges. * setgid is a process when executed will run as the group that owns the file.
What is the purpose of sticky bit?
Its a bit that only allows the owner or the root user to delete or modify the file.
What the following commands do? - chmod - chown - chgrp
* chmod - changes access permissions to files system objects * chown - changes the owner of file system files and directories * chgrp - changes the group associated with a file system object
What is sudo? How do you set it up?
True or False? In order to install packages on the system one must be the root user or use the sudo command
True
Explain what are ACLs. For what use cases would you recommend to use them?
You try to create a file but it fails. Name at least three different reason as to why it could happen
* No more disk space * No more inodes * No permissions
A user accidentally executed the following chmod -x $(which chmod). How to fix it?
Using `sudo setfacl -m u::rx /usr/bin/chmod` will set the execute permissions on `chmod` for all the users. Post this, the `chmod` binary can be used as usual.
You would like to copy a file to a remote Linux host. How would you do?
There are multiple ways to transfer files between hosts. Personal opinion: use `rsync`
How to generate a random string?
One way is to run the following: `cat /proc/sys/kernel/random/uuid`
How to generate a random string of 7 characters?
`mkpasswd -l 7`
What is systemd?
Systemd is a daemon (System 'd', d stands for daemon). A daemon is a program that runs in the background without direct control of the user, although the user can at any time talk to the daemon. systemd has many features such as user processes control/tracking, snapshot support, inhibitor locks.. If we visualize the unix/linux system in layers, systemd would fall directly after the linux kernel.
Hardware -> Kernel -> Daemons, System Libraries, Server Display.
How to start or stop a service?
To start a service: `systemctl start
How to check the status of a service?
`systemctl status
On a system which uses systemd, how would you display the logs?
journalctl
Describe how to make a certain process/app a service
Where system logs are located?
/var/log
How to follow file's content as it being appended without opening the file every time?
tail -f
What are you using for troubleshooting and debugging network issues?
dstat -t is great for identifying network and disk issues.
netstat -tnlaup can be used to see which processes are running on which ports.
lsof -i -P can be used for the same purpose as netstat.
ngrep -d any metafilter for matching regex against payloads of packets.
tcpdump for capturing packets
wireshark same concept as tcpdump but with GUI (optional).
What are you using for troubleshooting and debugging disk & file system issues?
dstat -t is great for identifying network and disk issues.
opensnoop can be used to see which files are being opened on the system (in real time).
What are you using for troubleshooting and debugging process issues?
strace is great for understanding what your program does. It prints every system call your program executed.
What are you using for debugging CPU related issues?
top will show you how much CPU percentage each process consumes
perf is a great choice for sampling profiler and in general, figuring out what your CPU cycles are "wasted" on
flamegraphs is great for CPU consumption visualization (http://www.brendangregg.com/flamegraphs.html)
You get a call from someone claiming "my system is SLOW". What do you do?
* Check with `top` for anything unusual * Run `dstat -t` to check if it's related to disk or network. * Check if it's network related with `sar` * Check I/O stats with `iostat`
Explain iostat output
How to debug binaries?
What is the difference between CPU load and utilization?
How you measure time execution of a program?
You have a process writing to a file. You don't know which process exactly, you just know the path of the file. You would like to kill the process as it's no longer needed. How would you achieve it?
1. Run `lsof
What is a kernel, and what does it do?
The kernel is part of the operating system and is responsible for tasks like: * Allocating memory * Schedule processes * Control CPU
How do you find out which Kernel version your system is using?
`uname -a` command
What is a Linux kernel module and how do you load a new module?
Explain user space vs. kernel space
The operating system executes the kernel in protected memory to prevent anyone from changing (and risking it crashing). This is what is known as "Kernel space". "User space" is where users executes their commands or applications. It's important to create this separation since we can't rely on user applications to not tamper with the kernel, causing it to crash. Applications can access system resources and indirectly the kernel space by making what is called "system calls".
In what phases of kernel lifecycle, can you change its configuration?
* Build time (when it's compiled) * Boot time (when it starts) * Runtime (once it's already running)
Where can you find kernel's configuration?
Usually it will reside in `/boot/config-
Where can you find the file that contains the command passed to the boot loader to run the kernel?
`/proc/cmdline`
How to list kernel's runtime parameters?
`sysctl -a`
Will running sysctl -a as a regular user vs. root, produce different result?
Yes, you might notice that in most systems, when running `systctl -a` with root, you'll get more runtime parameters compared to executing the same command with a regular user.
You would like to enable IPv4 forwarding in the kernel, how would you do it?
`sudo sysctl net.ipv4.ip_forward=1` To make it persistent (applied after reboot for example): insert `net.ipv4.ip_forward = 1` into `/etc/sysctl.conf` Another way to is to run `echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward`
How sysctl applies the changes to kernel's runtime parameters the moment you run sysctl command?
If you `strace` the sysctl command you can see it does it by changing the file under /proc/sys/... In the past it was done with sysctl system call, but it was deprecated at some point.
How changes to kernel runtime parameters persist? (applied even after reboot to the system for example)
There is a service called `systemd-sysctl` that takes the content of /etc/sysctl.conf and applies it. This is how changes persist, even after reboot, when they are written in /etc/sysctl.conf
Are the changes you make to kernel parameters in a container, affects also the kernel parameters of the host on which the container runs?
No. Containers have their own /proc filesystem so any change to kernel parameters inside a container, are not affecting the host or other containers running on that host.
What is SSH? How to check if a Linux server is running SSH?
[Wikipedia Definition](https://en.wikipedia.org/wiki/SSH_(Secure_Shell)): "SSH or Secure Shell is a cryptographic network protocol for operating network services securely over an unsecured network." [Hostinger.com Definition](https://www.hostinger.com/tutorials/ssh-tutorial-how-does-ssh-work): "SSH, or Secure Shell, is a remote administration protocol that allows users to control and modify their remote servers over the Internet." An SSH server will have SSH daemon running. Depends on the distribution, you should be able to check whether the service is running (e.g. systemctl status sshd).
Why SSH is considered better than telnet?
Telnet also allows you to connect to a remote host but as opposed to SSH where the communication is encrypted, in telnet, the data is sent in clear text, so it doesn't considered to be secured because anyone on the network can see what exactly is sent, including passwords.
What is stored in ~/.ssh/known_hosts?
The file stores the key fingerprints for the clients connecting to the SSH server. This fingerprint creates a trust between the client and the server for future SSH connections.
You try to ssh to a server and you get "Host key verification failed". What does it mean?
It means that the key of the remote host was changed and doesn't match the one that stored on the machine (in ~/.ssh/known_hosts).
What is the difference between SSH and SSL?
What ssh-keygen is used for?
ssh-keygen is a tool to generate an authentication key pair for SSH, that consists of a private and a public key. It supports a number of algorithms to generate authentication keys :
- dsa
- ecdsa
- ecdsa-sk
- ed25519
- ed25519-sk
- rsa (default)
One can also specify number of bits in key. Command below generates an SSH key pair with RSA 4096-bits :
```
$ ssh-keygen -t rsa -b 4096
```
The output looks like this:
```
Generating public/private rsa key pair.
Enter file in which to save the key (/home/user/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/user/.ssh/id_rsa
Your public key has been saved in /home/user/.ssh/id_rsa.pub
The key fingerprint is:
SHA256:f5MOGnhzYfC0ZCHvbSXXiRiNVYETjxpHcXD5xSojx+M user@mac-book-pro
The key's randomart image is:
+---[RSA 4096]----+
| . ..+***o|
| o o++*o+|
| . =+.++++|
| B.oX+. .|
| S *=o+ |
| . o oE. |
| . + + + |
| . = + . |
| . . |
+----[SHA256]-----+
```
One can check how many bits an SSH key has with :
```
$ ssh-keygen -l -f /home/user/.ssh/id_rsa
```
Output should look like this :
```
4096 SHA256:f5MOGnhzYfC0ZCHvbSXXiRiNVYETjxpHcXD5xSojx+M user@mac-book-pro (RSA)
```
It shows the key is RSA 4096-bits.
`-l` and `-f` parameters usage explanation :
```
-l Show the fingerprint of the key file.
-f filename Filename of the key file.
```
Learn more : [How can I tell how many bits my ssh key is? - Superuser](https://superuser.com/a/139311)
What is SSH port forwarding?
What is Globbing?
What are wildcards? Can you give an example of how to use them?
Explain what will ls [XYZ] match
Explain what will ls [^XYZ] match
Explain what will ls [0-5] match
What each of the following matches - ? - *
* The ? matches any single character * The * matches zero or more characters
What do we grep for in each of the following commands?:
* grep '[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}' some_file
* grep -E "error|failure" some_file
* grep '[0-9]$' some_file
1. An IP address 2. The word "error" or "failure" 3. Lines which end with a number
Which line numbers will be printed when running `grep '\baaa\b'` on the following content: aaa bbb ccc.aaa aaaaaa
lines 1 and 3.
What is the difference single and double quotes?
What is escaping? What escape character is used for escaping?
What is an exit code? What exit codes are you familiar with?
An exit code (or return code) represents the code returned by a child process to its parent process. 0 is an exit code which represents success while anything higher than 1 represents error. Each number has different meaning, based on how the application was developed. I consider this as a good blog post to read more about it: https://shapeshed.com/unix-exit-codes
Tell me everything you know about the Linux boot process
Another way to ask this: what happens from the moment you turned on the server until you get a prompt
What is GRUB2?
What is Secure Boot?
What can you find in /boot?
What's an inode?
For each file (and directory) in Linux there is an inode, a data structure which stores meta data related to the file like its size, owner, permissions, etc.
Which of the following is not included in inode: * Link count * File size * File name * File timestamp
File name (it's part of the directory file)
How to check which disks are currently mounted?
Run `mount`
You run the mount command but you get no output. How would you check what mounts you have on your system?
`cat /proc/mounts`
What is the difference between a soft link and hard link?
Hard link is the same file, using the same inode. Soft link is a shortcut to another file, using a different inode.
True or False? You can create an hard link for a directory
False
True or False? You can create a soft link between different filesystems
True
True or False? Directories always have by minimum 2 links
True.
What happens when you delete the original file in case of soft link and hard link?
Can you check what type of filesystem is used in /home?
There are many answers for this question. One way is running `df -T`
What is a swap partition? What is it used for?
How to create a - new empty file - a file with text (without using text editor) - a file with given size
* touch new_file.txt * cat > new_file [enter] submit text; ctrl + d to exit insert mode * truncate -s
You are trying to create a new file but you get "File system is full". You check with df for free space and you see you used only 20% of the space. What could be the problem?
How would you check what is the size of a certain directory?
`du -sh`
What is LVM?
Explain the following in regards to LVM: * PV * VG * LV
What is NFS? What is it used for?
What RAID is used for? Can you explain the differences between RAID 0, 1, 5 and 10?
Describe the process of extending a filesystem disk space
What is lazy umount?
What is tmpfs?
What is stored in each of the following logs? * /var/log/messages * /var/log/boot.log
True or False? both /tmp and /var/tmp cleared upon system boot
False. /tmp is cleared upon system boot while /var/tmp is cleared every a couple of days or not cleared at all (depends on distro).
How to check what is the current load average?
One can use `uptime` or `top`
You know how to see the load average, great. but what each part of it means? for example 1.43, 2.34, 2.78
[This article](http://www.brendangregg.com/blog/2017-08-08/linux-load-averages.html) summarizes the load average topic in a great way
How to check process usage?
pidstat
How to check disk I/O?
`iostat -xz 1`
How to check how much free memory a system has? How to check memory consumption by each process?
You can use the commands
top and free
How to check TCP stats?
sar -n TCP,ETCP 1
how to list all the processes running in your system?
The "ps" command can be used to list all the processes running in a system. The "ps aux" command provides a detailed list of all the processes, including the ones running in the background.
How to run a process in the background and why to do that in the first place?
You can achieve that by specifying & at the end of the command. As to why, since some commands/processes can take a lot of time to finish execution or run forever, you may want to run them in the background instead of waiting for them to finish before gaining control again in current session.
How can you find how much memory a specific process consumes?
mem()
{
ps -eo rss,pid,euser,args:100 --sort %mem | grep -v grep | grep -i $@ | awk '{printf $1/1024 "MB"; $1=""; print }'
}
[Source](https://stackoverflow.com/questions/3853655/in-linux-how-to-tell-how-much-memory-processes-are-using)
What signal is used by default when you run 'kill *process id*'?
The default signal is SIGTERM (15). This signal kills process gracefully which means it allows it to save current state configuration.
What signals are you familiar with?
SIGTERM - default signal for terminating a process SIGHUP - common usage is for reloading configuration SIGKILL - a signal which cannot caught or ignored To view all available signals run `kill -l`
What kill 0 does?
"kill 0" sends a signal to all processes in the current process group. It is used to check if the processes exist or not
What kill -0 </code> does?</summary>
"kill -0" checks if a process with a given process ID exists or not. It does not actually send any signal to the process.
</details>
What is a trap?
A trap is a mechanism that allows the shell to intercept signals sent to a process and perform a specific action, such as handling errors or cleaning up resources before terminating the process.
Every couple of days, a certain process stops running. How can you look into why it's happening?
One way to investigate why a process stops running is to check the system logs, such as the messages in /var/log/messages or journalctl. Additionally, checking the process's resource usage and system load may provide clues as to what caused the process to stop
What happens when you press ctrl + c?
When you press "Ctrl+C," it sends the SIGINT signal to the foreground process, asking it to terminate gracefully.
What is a Daemon in Linux?
A background process. Most of these processes are waiting for requests or set of conditions to be met before actually running anything.
Some examples: sshd, crond, rpcbind.
What are the possible states of a process in Linux?
Running (R)
Uninterruptible Sleep (D) - The process is waiting for I/O
Interruptible Sleep (S)
Stopped (T)
Dead (x)
Zombie (z)
How do you kill a process in D state?
A process in D state (also known as "uninterruptible sleep") cannot be killed using the "kill" command. The only way to terminate it is to reboot the system.
What is a zombie process?
A process which has finished to run but has not exited.
One reason it happens is when a parent process is programmed incorrectly. Every parent process should execute wait() to get the exit code from the child process which finished to run. But when the parent isn't checking for the child exit code, the child process can still exists although it finished to run.
How to get rid of zombie processes?
You can't kill a zombie process the regular way with `kill -9` for example as it's already dead.
One way to kill zombie process is by sending SIGCHLD to the parent process telling it to terminate its child processes. This might not work if the parent process wasn't programmed properly. The invocation is `kill -s SIGCHLD [parent_pid]`
You can also try closing/terminating the parent process. This will make the zombie process a child of init (1) which does periodic cleanups and will at some point clean up the zombie process.
How to find all the
* Processes executed/owned by a certain user
* Process which are Java processes
* Zombie Processes
If you mention at any point ps command with arguments, be familiar with what these arguments does exactly.
What is the init process?
It is the first process executed by the kernel during the booting of a system. It is a daemon process which runs till the system is shutdown. That is why, it is the parent of all the processes
Can you describe how processes are being created?
How to change the priority of a process? Why would you want to do that?
To change the priority of a process, you can use the nice command in Linux. The nice command allows you to specify the priority of a process by assigning a priority value ranging from -20 to 19. A higher value of priority means lower priority for the process, and vice versa.
You may want to change the priority of a process to adjust the amount of CPU time it is allocated by the system scheduler. For example, if you have a CPU-intensive process running on your system that is slowing down other processes, you can lower its priority to give more CPU time to other processes.
Can you explain how network process/connection is established and how it's terminated?>
</b>
When a client process on one system wants to establish a connection with a server process on another system, it first creates a socket using the socket system call. The client then calls the connect system call, passing the address of the server as an argument. This causes a three-way handshake to occur between the client and server, where the two systems exchange information to establish a connection.
Once the connection is established, the client and server can exchange data using the read and write system calls. When the connection is no longer needed, the client or server can terminate the connection by calling the close system call on the socket.
</b>
What strace does? What about ltrace?
Strace is a debugging tool that is used to monitor the system calls made by a process. It allows you to trace the execution of a process and see the system calls it makes, as well as the signals it receives. This can be useful for diagnosing issues with a process, such as identifying why it is hanging or crashing.
Ltrace, on the other hand, is a similar tool that is used to trace the library calls made by a process. It allows you to see the function calls made by a process to shared libraries, as well as the arguments passed to those functions. This can be useful for diagnosing issues with a process that involve library calls, such as identifying why a particular library is causing a problem.
Find all the files which end with '.yml' and replace the number 1 in 2 in each file
find /some_dir -iname \*.yml -print0 | xargs -0 -r sed -i "s/1/2/g"
You run ls and you get "/lib/ld-linux-armhf.so.3 no such file or directory". What is the problem?
The ls executable is built for an incompatible architecture.
How would you split a 50 lines file into 2 files of 25 lines each?
You can use the split command this way: split -l 25 some_file
What is a file descriptor? What file descriptors are you familiar with?
Kerberos
File descriptor, also known as file handler, is a unique number which identifies an open file in the operating system.
In Linux (and Unix) the first three file descriptors are:
* 0 - the default data stream for input
* 1 - the default data stream for output
* 2 - the default data stream for output related to errors
This is a great article on the topic: https://www.computerhope.com/jargon/f/file-descriptor.htm
What is NTP? What is it used for?
Explain Kernel OOM
### Security
What is chroot? In what scenarios would you consider using it?
What is SELiunx?
What is Kerberos?
What is nftables?
What firewalld daemon is responsible for?
Do you have experience with hardening servers? Can you describe the process?
How do you create a private key for a CA (certificate authority)?
One way is using openssl this way:
`openssl genrsa -aes256 -out ca-private-key.pem 4096`
How do you create a public key for a CA (certificate authority)?
`openssl req -new -x509 -days 730 -key [private key file name] -sha256 -out ca.pem`
If using the private key from the previous question then the command would be:
`openssl req -new -x509 -days 730 -key ca-private-key.pem -sha256 -out ca.pem`
Demonstrate one way to encode and decode data in Linux
Encode: `echo -n "some password" | base64`
Decode: `echo -n "allE19remO91" | base64`
### Networking
How to list all the interfaces?
```
ip link show
```
What is the loopback (lo) interface?
The loopback interface is a special, virtual network interface that your computer uses to communicate with itself. It is used mainly for diagnostics and troubleshooting, and to connect to servers running on the local machine.
What the following commands are used for?
* ip addr
* ip route
* ip link
* ping
* netstat
* traceroute
What is a network namespace? What is it used for?
How to check if a certain port is being used?
One of the following would work:
```
netstat -tnlp | grep
lsof -i -n -P | grep
```
</b></details>
How can you turn your Linux server into a router?
What is a virtual IP? In what situation would you use it?
True or False? The MAC address of an interface is assigned/set by the OS
False
Can you have more than one default gateway in a given system?
Technically, yes.
What is telnet and why is it a bad idea to use it in production? (or at all)
Telnet is a type of client-server protocol that can be used to open a command line on a remote computer, typically a server.
By default, all the data sent and received via telnet is transmitted in clear/plain text, therefore it should not be used as it does not encrypt any data between the client and the server.
What is the routing table? How do you view it?
How can you send an HTTP request from your shell?
Using nc is one way
What are packet sniffers? Have you used one in the past? If yes, which packet sniffers have you used and for what purpose?
It is a network utility that analyses and may inject tasks into the data-stream travelling over the targeted network.
How to list active connections?
How to trigger neighbor discovery in IPv6?
One way would be `ping6 ff02::1`
What is network interface bonding and do you know how it's performed in Linux?
What network bonding modes are there?
There a couple of modes:
* balance-rr: round robing bonding
* active-backup: a fault tolerance mode where only one is active
* balance-tlb: Adaptive transmit load balancing
* balance-alb: Adaptive load balancing
What is a bridge? How it's added in Linux OS?
### DNS
How to check what is the hostname of the system?
`cat /etc/hostname`
You can also run `hostnamectl` or `hostname` but that might print only a temporary hostname. The one in the file is the permanent one.
What the file /etc/resolv.conf is used for? What does it include?
What commands are you using for performing DNS queries (or troubleshoot DNS related issues)?
You can specify one or more of the following:
* dig
* host
* nslookup
You run dig codingshell.com and get the following result:
```
ANSWER SECTION:
codingshell.com. 3515 IN A 185.199.109.153
```
What is the meaning of the number 3515?
This is the TTL. When you lookup for an address using a domain/host name, your OS is performing DNS resolution by contacting DNS name servers to get the IP address of the host/domain you are looking for.
When you get a reply, this reply in cached in your OS for a certain period of time. This is period of time is also known as TTL and this is the meaning of 3515 number - it will be cached for 3515 seconds before removed from the cache and during that period of time, you'll get the value from the cache instead of asking DNS name servers for the address again.
How can we modify the network connection via `nmcli` command, to use `8.8.8.8` as a DNS server?
1. Find the connection name:
```
# nmcli con show
NAME UUID TYPE DEVICE
System ens5 8126c120-a964-e959-ff98-ac4973344505 ethernet ens5
System eth0 5fb06bd0-0bb0-7ffb-45f1-d6edd65f3e03 ethernet --
```
Here the connection name is "System ens5". Let's say we want to modify settings for this connection.
2. Modify the connection to use 8.8.8.8 as DNS server:
```
# nmcli con mod "System ens5" ipv4.dns "8.8.8.8"
```
3. We need to reactivate the connection for the change to take effect:
```
nmcli con up "System ens5"
```
4. Verify our settings once more:
```
cat /etc/resolv.conf
nmcli -f ipv4.dns con show "System ens5"
```
### Packaging
Do you have experience with packaging? (as in building packages) Can you explain how does it works?
How packages installation/removal is performed on the distribution you are using?
The answer depends on the distribution being used.
In Fedora/CentOS/RHEL/Rocky it can be done with `rpm` or `dnf` commands.
In Ubuntu it can be done with the `apt` command.
RPM: explain the spec format (what it should and can include)
How do you list the content of a package without actually installing it?
How to know to which package a file on the system belongs to? Is it a problem if it doesn't belongs to any package?
Where repositories are stored? (based on the distribution you are using)
What is an archive? How do you create one in Linux?
How to extract the content of an archive?
Why do we need package managers? Why not simply creating archives and publish them?
Package managers allow you to manage packages lifecycle as in installing, removing and updating the packages.
In addition, you can specify in a spec how a certain package will be installed - where to copy the files, which commands to run prior to the installation, post the installation, etc.
### DNF
What is DNF?
From the [repo](https://github.com/rpm-software-management/dnf):
"Dandified YUM (DNF) is the next upcoming major version of YUM. It does package management using RPM, libsolv and hawkey libraries."
Official [docs](https://dnf.readthedocs.io/en/latest/)
How to look for a package that provides the command /usr/bin/git? (the package isn't necessarily installed)
dnf provides /usr/bin/git
### Applications and Services
What can you find in /etc/services?
How to make sure a Service starts automatically after a reboot or crash?
Depends on the init system.
Systemd: systemctl enable [service_name]
System V: update-rc.d [service_name] and add this line id:5678:respawn:/bin/sh /path/to/app to /etc/inittab
Upstart: add Upstart init script at /etc/init/service.conf
You run ssh 127.0.0.1 but it fails with "connection refused". What could be the problem?
1. SSH server is not installed
2. SSH server is not running
How to print the shared libraries required by a certain program? What is it useful for?
What is CUPS?
What types of web servers are you familiar with?
Nginx, Apache httpd.
### Users and Groups
What is a "superuser" (or root user)? How is it different from regular users?
How do you create users? Where user information is stored?
Command to create users is `useradd`
Syntax:
`useradd [options] Username`
There are 2 configuration files, which stores users information
1. `/etc/passwd` - Users information like, username, shell etc is stored in this file
2. `/etc/shadow` - Users password is stored in encrypted format
Which file stores information about groups?
`/etc/groups` file stores the group name, group ID, usernames which are in secondary group.
How do you change/set the password of a user?
`passwd ` is the command to set/change password of a user.
</details>
Which file stores users passwords? Is it visible for everyone?
`/etc/shadow` file holds the passwords of the users in encryted format. NO, it is only visible to the `root` user
Do you know how to create a new user without using adduser/useradd command?
YES, we can create new user by manually adding an entry in the `/etc/passwd` file.
For example, if we need to create a user called `john`.
Step 1: Add an entry to `/etc/passwd` file, so user gets created.
`echo "john:x:2001:2001::/home/john:/bin/bash" >> /etc/passwd`
Step 2: Add an entry to `/etc/group` file, because every user belong to the primary group that has same name as the username.
`echo "john:x:2001:" >> /etc/group`
Step 3: Verify if the user got created
`id john`
What information is stored in /etc/passwd? explain each field
`/etc/passwd` is a configuration file, which contains users information. Each entry in this file has, 7 fields,
`username:password:UID:GID:Comment:home directory:shell`
`username` - The name of the user.
`password` - This field is actually a placeholder of the password field. Due to security concerns, this field does not contain the password, just a placeholder (x) to the encrypted password stored in `/etc/shadow` file.
`UID` - User ID of the user.
`GID` - Group ID
`Comment` - This field is to provide description about the user.
`home directory` - Abousulte path of the user's home directory. This directory gets created once the user is added.
`shell` - This field contains the absolute path of the shell that will be used by the respective user.
How to add a new user to the system without providing him the ability to log-in into the system?
`adduser user_name --shell=/bin/false --no-create-home`
You can also add a user and then edit /etc/passwd.
How to switch to another user? How to switch to the root user?
su command.
Use su - to switch to root
What is the UID the root user? What about a regular user?
UID of root user is 0
Default values of UID_MIN and UID_MAX in `/etc/login.defs`
`UID_MIN` is `1000`
`UID_MAX` is `60000`
Actually, we can change this value. But UID < 1000 are reserved for system accounts.
Therefore, as per the default configuration, for regular user UID starts from `1000`.
What can you do if you lost/forogt the root password?
Re-install the OS IS NOT the right answer :)
What is /etc/skel?
`/etc/skel` is a directory, that contains files or directories, so when a new user is created, these files/directories created under `/etc/skel` will be copied to user's home directory.
How to see a list of who logged-in to the system?
Using the `last` command.
Explain what each of the following commands does:
* useradd
* usermod
* whoami
* id
`useradd` - Command for creating new users
`usermod` - Modify the users setting
`whoami` - Outputs, the username that we are currently logged in
`id` - Prints the
You run grep $(whoami) /etc/passwd but the output is empty. What might be a possible reason for that?
The user you are using isn't defined locally but originates from services like LDAP.
You can verify with: `getent passwd`
### Hardware
Where can you find information on the processor (like number of CPUs)?
/proc/cpuinfo
You can also use `nproc` for number of processors
How can you print information on the BIOS, motherboard, processor and RAM?
dmidecoode
How can you print all the information on connected block devices in your system?
lsblk
True or False? In user space, applications don't have full access to hardware resources
True. Only in kernel space they have full access to hardware resources.
### Namespaces
What types of namespaces are there in Linux?
- Process ID namespaces: these namespaces include independent set of process IDs
- Mount namespaces: Isolation and control of mountpoints
- Network namespaces: Isolates system networking resources such as routing table, interfaces, ARP table, etc.
- UTS namespaces: Isolate host and domains
- IPC namespaces: Isolates interprocess communications
- User namespaces: Isolate user and group IDs
- Time namespaces: Isolates time machine
True or False? In every PID (Process ID) namespace the first process assigned with the process id number 1
True. Inside the namespace it's PID 1 while to the parent namespace the PID is a different one.
True or False? In a child PID namespace all processes are aware of parent PID namespace and processes and the parent PID namespace has no visibility of child PID namespace processes
False. The opposite is true. Parent PID namespace is aware and has visibility of processes in child PID namespace and child PID namespace has no visibility as to what is going on in the parent PID namespace.
True or False? By default, when creating two separate network namespaces, a ping from one namespace to another will work fine
False. Network namespace has its own interfaces and routing table. There is no way (without creating a bridge for example) for one network namespace to reach another.
True or False? With UTS namespaces, processes may appear as if they run on different hosts and domains while running on the same host
True
True or False? It's not possible to have a root user with ID 0 in child user namespaces
False. In every child user namespace, it's possible to have a separate root user with uid of 0.
What time namespaces are used for?
In time namespaces processes can use different system time.
### Virtualization
What virtualization solutions are available for Linux?
* [KVM](https://www.linux-kvm.org/page/Main_Page)
* [XEN](http://www.xen.org/)
* [VirtualBox](https://www.virtualbox.org/)
* [Linux-VServer](http://linux-vserver.org/Welcome_to_Linux-VServer.org)
* [User-mode Linux](http://user-mode-linux.sourceforge.net/)
* ...
What is KVM?
Is an open source virtualization technology used to operate on x86 hardware.
From the official [docs](https://www.linux-kvm.org/page/Main_Page)
Recommended read:
* [Red Hat Article - What is KVM?](https://www.redhat.com/en/topics/virtualization/what-is-KVM)
What is Libvirt?
It's an open source collection of software used to manage virtual machines. It can be used with: KVM, Xen, LXC and others. It's also called Libvirt Virtualization API.
From the official [docs](https://libvirt.org/)
Hypervisor supported [docs](https://libvirt.org/drivers.html)
### AWK
What the awk command does? Have you used it? What for?
From Wikipedia: "AWK is domain-specific language designed for text processing and typically used as a data extraction and reporting tool"
How to print the 4th column in a file?
`awk '{print $4}' file`
How to print every line that is longer than 79 characters?
`awk 'length($0) > 79' file`
What the lsof command does? Have you used it? What for?
What is the difference between find and locate?
How a user process performs a privileged operation, such as reading from the disk?
Using system calls
### System Calls
What is a system call? What system calls are you familiar with?
How a program executes a system call?
- A program executes a trap instruction. The instruction jump into the kernel while raising the privileged level to kernel space.
- Once in kernel space, it can perform any privileged operation
- Once it's finished, it calls a "return-from-trap" instruction which returns to user space while reducing back the privilege level to user space.
Explain the fork() system call
fork() is used for creating a new process. It does so by cloning the calling process but the child process has its own PID and any memory locks, I/O operations and semaphores are not inherited.
What is the return value of fork()?
- On success, the PID of the child process in parent and 0 in child process
- On error, -1 in the parent
Name one reason for fork() to fail
Not enough memory to create a new process
Why do we need the wait() system call?
wait() is used by a parent process to wait for the child process to finish execution.
If wait is not used by a parent process then a child process might become a zombie process.
How the kernel notifies the parent process about child process termination?
The kernel notifies the parent by sending the SIGCHLD to the parent.
How the waitpid() is different from wait()?
The waitpid() is a non-blocking version of the wait() function.
It also supports using library routine (e.g. system()) to wait a child process without messing up with other children processes for which the process has not waited.
True or False? The wait() system call won't return until the child process has run and exited
True in most cases though there are cases where wait() returns before the child exits.
Explain the exec() system call
It transforms the current running program into another program.
Given the name of an executable and some arguments, it loads the code and static data from the specified executable and overwrites its current code segment and current static code data. After initializing its memory space (like stack and heap) the OS runs the program passing any arguments as the argv of that process.
True or False? A successful call to exec() never returns
True
Since a successful exec replace the current process, it can't return anything to the process that made the call.
What system call is used for listing files?
What system calls are used for creating a new process?
fork(), exec() and the wait() system call is also included in this workflow.
What execve() does?
Executes a program. The program is passed as a filename (or path) and must be a binary executable or a script.
What is the return value of malloc?
Explain the pipe() system call. What does it used for?
[Unix pipe implementation](https://toroid.org/unix-pipe-implementation)
"Pipes provide a unidirectional interprocess communication channel. A pipe has a read end and a write end. Data written to the write end of a pipe can be read from the read end of the pipe.
A pipe is created using pipe(2), which returns two file descriptors, one referring to the read end of the pipe, the other referring to the write end."
What happens when you execute ls -l?
* Shell reads the input using getline() which reads the input file stream and stores into a buffer as a string
* The buffer is broken down into tokens and stored in an array this way: {"ls", "-l", "NULL"}
* Shell checks if an expansion is required (in case of ls *.c)
* Once the program in memory, its execution starts. First by calling readdir()
Notes:
* getline() originates in GNU C library and used to read lines from input stream and stores those lines in the buffer
What happens when you execute ls -l *.log?
What readdir() system call does?
What exactly the command alias x=y does?
Why running a new program is done using the fork() and exec() system calls? why a different API wasn't developed where there is one call to run a new program?
This way provides a lot of flexibility. It allows the shell for example, to run code after the call to fork() but before the call to exec(). Such code can be used to alter the environment of the program it about to run.
Describe shortly what happens when you execute a command in the shell
The shell figures out, using the PATH variable, where the executable of the command resides in the filesystem. It then calls fork() to create a new child process for running the command. Once the fork was executed successfully, it calls a variant of exec() to execute the command and finally, waits the command to finish using wait(). When the child completes, the shell returns from wait() and prints out the prompt again.
### Filesystem & Files
How to create a file of a certain size?
There are a couple of ways to do that:
* dd if=/dev/urandom of=new_file.txt bs=2MB count=1
* truncate -s 2M new_file.txt
* fallocate -l 2097152 new_file.txt
What does the following block do?:
```
open("/my/file") = 5
read(5, "file content")
```
These system calls are reading the file /my/file and 5 is the file descriptor number.
Describe three different ways to remove a file (or its content)
What is the difference between a process and a thread?
What is context switch?
From [wikipedia](https://en.wikipedia.org/wiki/Context_switch): a context switch is the process of storing the state of a process or thread, so that it can be restored and resume execution at a later point
You found there is a server with high CPU load but you didn't find a process with high CPU. How is that possible?
### Advanced Networking
When you run ip a you see there is a device called 'lo'. What is it and why do we need it?
What the traceroute command does? How does it works?
Another common way to task this questions is "what part of the tcp header does traceroute modify?"
What is network bonding? What types are you familiar with?
How to link two separate network namespaces so you can ping an interface on one namespace from the second one?
What are cgroups?
Explain Process Descriptor and Task Structure
What are the differences between threads and processes?
Explain Kernel Threads
What happens when socket system call is used?
This is a good article about the topic: https://ops.tips/blog/how-linux-creates-sockets
You executed a script and while still running, it got accidentally removed. Is it possible to restore the script while it's still running?
### Memory
What is the difference between MemFree and MemAvailable in /proc/meminfo?
MemFree - The amount of unused physical RAM in your system
MemAvailable - The amount of available memory for new workloads (without pushing system to use swap) based on MemFree, Active(file), Inactive(file), and SReclaimable.
What is the difference between paging and swapping?
Explain what is OOM killer
### Distributions
What is a Linux distribution?
What Linux distributions are you familiar with?
What are the components of a Linux distribution?
* Kernel
* Utilities
* Services
* Software/Packages Management
### Sed
Using sed, extract the date from the following line: 201.7.19.90 - - [05/Jun/1985:13:42:99 +0000] "GET /site HTTP/1.1" 200 32421
`echo $line | sed 's/.*\[//g;s/].*//g;s/:.*//g'`
### Misc
What is a Linux distribution?
* A collection of packages - kernel, GNU, third party apps, ...
* Sometimes distributions store some information on the distribution in `/etc/*-release` file
* For example for Red Hat distribution it will be `/etc/redhat-release` and for Amazon it will be `/etc/os-release`
* `lsb_release` is a common command you can use in multiple different distributions
Name 5 commands which are two letters long
ls, wc, dd, df, du, ps, ip, cp, cd ...
What ways are there for creating a new empty file?
* touch new_file
* echo "" > new_file
How `cd -` works? How does it knows the previous location?
$OLDPWD
List three ways to print all the files in the current directory
* ls
* find .
* echo *
How to count the number of lines in a file? What about words?
You define x=2 in /etc/bashrc and x=6 ~/.bashrc you then login to the system. What would be the value of x?
What is the difference between man and info?
A good answer can be found [here](https://askubuntu.com/questions/9325/what-is-the-difference-between-man-and-info-documentation)
Explain "environment variables". How do you list all environment variables?
What is a TTY device?
How to create your own environment variables?
`X=2` for example. But this will persist to new shells. To have it in new shells as well, use `export X=2`
What a double dash (--) mean?
It's used in commands to mark the end of commands options. One common example is when used with git to discard local changes: `git checkout -- some_file`
Wildcards are implemented on user or kernel space?
If I plug a new device into a Linux machine, where on the system, a new device entry/file will be created?
/dev
Why there are different sections in man? What is the difference between the sections?
What is User-mode Linux?
In Linux, user mode is a restricted operating mode in which a user's application or process runs. User mode is a non-privileged mode that prevents user-level processes from accessing sensitive system resources directly.
In user mode, an application can only access hardware resources indirectly, by calling system services or functions provided by the operating system. This ensures that the system's security and stability are maintained by preventing user processes from interfering with or damaging system resources.
Additionally, user mode also provides memory protection to prevent applications from accessing unauthorized memory locations. This is done by assigning each process its own virtual memory space, which is isolated from other processes.
In contrast to user mode, kernel mode is a privileged operating mode in which the operating system's kernel has full access to system resources, and can perform low-level operations, such as accessing hardware devices and managing system resources directly.
Under which license Linux is distributed?
GPL v2
"kill -0" checks if a process with a given process ID exists or not. It does not actually send any signal to the process. </details>
What is a trap?
A trap is a mechanism that allows the shell to intercept signals sent to a process and perform a specific action, such as handling errors or cleaning up resources before terminating the process.
Every couple of days, a certain process stops running. How can you look into why it's happening?
One way to investigate why a process stops running is to check the system logs, such as the messages in /var/log/messages or journalctl. Additionally, checking the process's resource usage and system load may provide clues as to what caused the process to stop
What happens when you press ctrl + c?
When you press "Ctrl+C," it sends the SIGINT signal to the foreground process, asking it to terminate gracefully.
What is a Daemon in Linux?
A background process. Most of these processes are waiting for requests or set of conditions to be met before actually running anything. Some examples: sshd, crond, rpcbind.
What are the possible states of a process in Linux?
Running (R) Uninterruptible Sleep (D) - The process is waiting for I/O Interruptible Sleep (S) Stopped (T) Dead (x) Zombie (z)
How do you kill a process in D state?
A process in D state (also known as "uninterruptible sleep") cannot be killed using the "kill" command. The only way to terminate it is to reboot the system.
What is a zombie process?
A process which has finished to run but has not exited. One reason it happens is when a parent process is programmed incorrectly. Every parent process should execute wait() to get the exit code from the child process which finished to run. But when the parent isn't checking for the child exit code, the child process can still exists although it finished to run.
How to get rid of zombie processes?
You can't kill a zombie process the regular way with `kill -9` for example as it's already dead. One way to kill zombie process is by sending SIGCHLD to the parent process telling it to terminate its child processes. This might not work if the parent process wasn't programmed properly. The invocation is `kill -s SIGCHLD [parent_pid]` You can also try closing/terminating the parent process. This will make the zombie process a child of init (1) which does periodic cleanups and will at some point clean up the zombie process.
How to find all the * Processes executed/owned by a certain user * Process which are Java processes * Zombie Processes
If you mention at any point ps command with arguments, be familiar with what these arguments does exactly.
What is the init process?
It is the first process executed by the kernel during the booting of a system. It is a daemon process which runs till the system is shutdown. That is why, it is the parent of all the processes
Can you describe how processes are being created?
How to change the priority of a process? Why would you want to do that?
To change the priority of a process, you can use the nice command in Linux. The nice command allows you to specify the priority of a process by assigning a priority value ranging from -20 to 19. A higher value of priority means lower priority for the process, and vice versa. You may want to change the priority of a process to adjust the amount of CPU time it is allocated by the system scheduler. For example, if you have a CPU-intensive process running on your system that is slowing down other processes, you can lower its priority to give more CPU time to other processes.
Can you explain how network process/connection is established and how it's terminated?>
</b> When a client process on one system wants to establish a connection with a server process on another system, it first creates a socket using the socket system call. The client then calls the connect system call, passing the address of the server as an argument. This causes a three-way handshake to occur between the client and server, where the two systems exchange information to establish a connection. Once the connection is established, the client and server can exchange data using the read and write system calls. When the connection is no longer needed, the client or server can terminate the connection by calling the close system call on the socket. </b>
What strace does? What about ltrace?
Strace is a debugging tool that is used to monitor the system calls made by a process. It allows you to trace the execution of a process and see the system calls it makes, as well as the signals it receives. This can be useful for diagnosing issues with a process, such as identifying why it is hanging or crashing. Ltrace, on the other hand, is a similar tool that is used to trace the library calls made by a process. It allows you to see the function calls made by a process to shared libraries, as well as the arguments passed to those functions. This can be useful for diagnosing issues with a process that involve library calls, such as identifying why a particular library is causing a problem.
Find all the files which end with '.yml' and replace the number 1 in 2 in each file
find /some_dir -iname \*.yml -print0 | xargs -0 -r sed -i "s/1/2/g"
You run ls and you get "/lib/ld-linux-armhf.so.3 no such file or directory". What is the problem?
The ls executable is built for an incompatible architecture.
How would you split a 50 lines file into 2 files of 25 lines each?
You can use the
split command this way: split -l 25 some_file
What is a file descriptor? What file descriptors are you familiar with?
Kerberos File descriptor, also known as file handler, is a unique number which identifies an open file in the operating system. In Linux (and Unix) the first three file descriptors are: * 0 - the default data stream for input * 1 - the default data stream for output * 2 - the default data stream for output related to errors This is a great article on the topic: https://www.computerhope.com/jargon/f/file-descriptor.htm
What is NTP? What is it used for?
Explain Kernel OOM
What is chroot? In what scenarios would you consider using it?
What is SELiunx?
What is Kerberos?
What is nftables?
What firewalld daemon is responsible for?
Do you have experience with hardening servers? Can you describe the process?
How do you create a private key for a CA (certificate authority)?
One way is using openssl this way: `openssl genrsa -aes256 -out ca-private-key.pem 4096`
How do you create a public key for a CA (certificate authority)?
`openssl req -new -x509 -days 730 -key [private key file name] -sha256 -out ca.pem` If using the private key from the previous question then the command would be: `openssl req -new -x509 -days 730 -key ca-private-key.pem -sha256 -out ca.pem`
Demonstrate one way to encode and decode data in Linux
Encode: `echo -n "some password" | base64` Decode: `echo -n "allE19remO91" | base64`
How to list all the interfaces?
``` ip link show ```
What is the loopback (lo) interface?
The loopback interface is a special, virtual network interface that your computer uses to communicate with itself. It is used mainly for diagnostics and troubleshooting, and to connect to servers running on the local machine.
What the following commands are used for? * ip addr * ip route * ip link * ping * netstat * traceroute
What is a network namespace? What is it used for?
How to check if a certain port is being used?
One of the following would work: ``` netstat -tnlp | grep
How can you turn your Linux server into a router?
What is a virtual IP? In what situation would you use it?
True or False? The MAC address of an interface is assigned/set by the OS
False
Can you have more than one default gateway in a given system?
Technically, yes.
What is telnet and why is it a bad idea to use it in production? (or at all)
Telnet is a type of client-server protocol that can be used to open a command line on a remote computer, typically a server. By default, all the data sent and received via telnet is transmitted in clear/plain text, therefore it should not be used as it does not encrypt any data between the client and the server.
What is the routing table? How do you view it?
How can you send an HTTP request from your shell?
Using nc is one way
What are packet sniffers? Have you used one in the past? If yes, which packet sniffers have you used and for what purpose?
It is a network utility that analyses and may inject tasks into the data-stream travelling over the targeted network.
How to list active connections?
How to trigger neighbor discovery in IPv6?
One way would be `ping6 ff02::1`
What is network interface bonding and do you know how it's performed in Linux?
What network bonding modes are there?
There a couple of modes: * balance-rr: round robing bonding * active-backup: a fault tolerance mode where only one is active * balance-tlb: Adaptive transmit load balancing * balance-alb: Adaptive load balancing
What is a bridge? How it's added in Linux OS?
How to check what is the hostname of the system?
`cat /etc/hostname` You can also run `hostnamectl` or `hostname` but that might print only a temporary hostname. The one in the file is the permanent one.
What the file /etc/resolv.conf is used for? What does it include?
What commands are you using for performing DNS queries (or troubleshoot DNS related issues)?
You can specify one or more of the following: *
dig
* host
* nslookup
You run dig codingshell.com and get the following result:
```
ANSWER SECTION:
codingshell.com. 3515 IN A 185.199.109.153
```
What is the meaning of the number 3515?
This is the TTL. When you lookup for an address using a domain/host name, your OS is performing DNS resolution by contacting DNS name servers to get the IP address of the host/domain you are looking for.
When you get a reply, this reply in cached in your OS for a certain period of time. This is period of time is also known as TTL and this is the meaning of 3515 number - it will be cached for 3515 seconds before removed from the cache and during that period of time, you'll get the value from the cache instead of asking DNS name servers for the address again.
How can we modify the network connection via `nmcli` command, to use `8.8.8.8` as a DNS server?
1. Find the connection name: ``` # nmcli con show NAME UUID TYPE DEVICE System ens5 8126c120-a964-e959-ff98-ac4973344505 ethernet ens5 System eth0 5fb06bd0-0bb0-7ffb-45f1-d6edd65f3e03 ethernet -- ``` Here the connection name is "System ens5". Let's say we want to modify settings for this connection. 2. Modify the connection to use 8.8.8.8 as DNS server: ``` # nmcli con mod "System ens5" ipv4.dns "8.8.8.8" ``` 3. We need to reactivate the connection for the change to take effect: ``` nmcli con up "System ens5" ``` 4. Verify our settings once more: ``` cat /etc/resolv.conf nmcli -f ipv4.dns con show "System ens5" ```
Do you have experience with packaging? (as in building packages) Can you explain how does it works?
How packages installation/removal is performed on the distribution you are using?
The answer depends on the distribution being used. In Fedora/CentOS/RHEL/Rocky it can be done with `rpm` or `dnf` commands. In Ubuntu it can be done with the `apt` command.
RPM: explain the spec format (what it should and can include)
How do you list the content of a package without actually installing it?
How to know to which package a file on the system belongs to? Is it a problem if it doesn't belongs to any package?
Where repositories are stored? (based on the distribution you are using)
What is an archive? How do you create one in Linux?
How to extract the content of an archive?
Why do we need package managers? Why not simply creating archives and publish them?
Package managers allow you to manage packages lifecycle as in installing, removing and updating the packages.
In addition, you can specify in a spec how a certain package will be installed - where to copy the files, which commands to run prior to the installation, post the installation, etc.
What is DNF?
From the [repo](https://github.com/rpm-software-management/dnf): "Dandified YUM (DNF) is the next upcoming major version of YUM. It does package management using RPM, libsolv and hawkey libraries." Official [docs](https://dnf.readthedocs.io/en/latest/)
How to look for a package that provides the command /usr/bin/git? (the package isn't necessarily installed)
dnf provides /usr/bin/git
What can you find in /etc/services?
How to make sure a Service starts automatically after a reboot or crash?
Depends on the init system. Systemd:
systemctl enable [service_name]
System V: update-rc.d [service_name] and add this line id:5678:respawn:/bin/sh /path/to/app to /etc/inittab
Upstart: add Upstart init script at /etc/init/service.conf
You run ssh 127.0.0.1 but it fails with "connection refused". What could be the problem?
1. SSH server is not installed 2. SSH server is not running
How to print the shared libraries required by a certain program? What is it useful for?
What is CUPS?
What types of web servers are you familiar with?
Nginx, Apache httpd.
What is a "superuser" (or root user)? How is it different from regular users?
How do you create users? Where user information is stored?
Command to create users is `useradd` Syntax: `useradd [options] Username` There are 2 configuration files, which stores users information 1. `/etc/passwd` - Users information like, username, shell etc is stored in this file 2. `/etc/shadow` - Users password is stored in encrypted format
Which file stores information about groups?
`/etc/groups` file stores the group name, group ID, usernames which are in secondary group.
How do you change/set the password of a user?
`passwd
Which file stores users passwords? Is it visible for everyone?
`/etc/shadow` file holds the passwords of the users in encryted format. NO, it is only visible to the `root` user
Do you know how to create a new user without using adduser/useradd command?
YES, we can create new user by manually adding an entry in the `/etc/passwd` file. For example, if we need to create a user called `john`. Step 1: Add an entry to `/etc/passwd` file, so user gets created. `echo "john:x:2001:2001::/home/john:/bin/bash" >> /etc/passwd` Step 2: Add an entry to `/etc/group` file, because every user belong to the primary group that has same name as the username. `echo "john:x:2001:" >> /etc/group` Step 3: Verify if the user got created `id john`
What information is stored in /etc/passwd? explain each field
`/etc/passwd` is a configuration file, which contains users information. Each entry in this file has, 7 fields, `username:password:UID:GID:Comment:home directory:shell` `username` - The name of the user. `password` - This field is actually a placeholder of the password field. Due to security concerns, this field does not contain the password, just a placeholder (x) to the encrypted password stored in `/etc/shadow` file. `UID` - User ID of the user. `GID` - Group ID `Comment` - This field is to provide description about the user. `home directory` - Abousulte path of the user's home directory. This directory gets created once the user is added. `shell` - This field contains the absolute path of the shell that will be used by the respective user.
How to add a new user to the system without providing him the ability to log-in into the system?
`adduser user_name --shell=/bin/false --no-create-home` You can also add a user and then edit /etc/passwd.
How to switch to another user? How to switch to the root user?
su command. Use su - to switch to root
What is the UID the root user? What about a regular user?
UID of root user is 0 Default values of UID_MIN and UID_MAX in `/etc/login.defs` `UID_MIN` is `1000` `UID_MAX` is `60000` Actually, we can change this value. But UID < 1000 are reserved for system accounts. Therefore, as per the default configuration, for regular user UID starts from `1000`.
What can you do if you lost/forogt the root password?
Re-install the OS IS NOT the right answer :)
What is /etc/skel?
`/etc/skel` is a directory, that contains files or directories, so when a new user is created, these files/directories created under `/etc/skel` will be copied to user's home directory.
How to see a list of who logged-in to the system?
Using the `last` command.
Explain what each of the following commands does: * useradd * usermod * whoami * id
`useradd` - Command for creating new users `usermod` - Modify the users setting `whoami` - Outputs, the username that we are currently logged in `id` - Prints the
You run grep $(whoami) /etc/passwd but the output is empty. What might be a possible reason for that?
The user you are using isn't defined locally but originates from services like LDAP.
You can verify with: `getent passwd`
Where can you find information on the processor (like number of CPUs)?
/proc/cpuinfo You can also use `nproc` for number of processors
How can you print information on the BIOS, motherboard, processor and RAM?
dmidecoode
How can you print all the information on connected block devices in your system?
lsblk
True or False? In user space, applications don't have full access to hardware resources
True. Only in kernel space they have full access to hardware resources.
What types of namespaces are there in Linux?
- Process ID namespaces: these namespaces include independent set of process IDs - Mount namespaces: Isolation and control of mountpoints - Network namespaces: Isolates system networking resources such as routing table, interfaces, ARP table, etc. - UTS namespaces: Isolate host and domains - IPC namespaces: Isolates interprocess communications - User namespaces: Isolate user and group IDs - Time namespaces: Isolates time machine
True or False? In every PID (Process ID) namespace the first process assigned with the process id number 1
True. Inside the namespace it's PID 1 while to the parent namespace the PID is a different one.
True or False? In a child PID namespace all processes are aware of parent PID namespace and processes and the parent PID namespace has no visibility of child PID namespace processes
False. The opposite is true. Parent PID namespace is aware and has visibility of processes in child PID namespace and child PID namespace has no visibility as to what is going on in the parent PID namespace.
True or False? By default, when creating two separate network namespaces, a ping from one namespace to another will work fine
False. Network namespace has its own interfaces and routing table. There is no way (without creating a bridge for example) for one network namespace to reach another.
True or False? With UTS namespaces, processes may appear as if they run on different hosts and domains while running on the same host
True
True or False? It's not possible to have a root user with ID 0 in child user namespaces
False. In every child user namespace, it's possible to have a separate root user with uid of 0.
What time namespaces are used for?
In time namespaces processes can use different system time.
What virtualization solutions are available for Linux?
* [KVM](https://www.linux-kvm.org/page/Main_Page) * [XEN](http://www.xen.org/) * [VirtualBox](https://www.virtualbox.org/) * [Linux-VServer](http://linux-vserver.org/Welcome_to_Linux-VServer.org) * [User-mode Linux](http://user-mode-linux.sourceforge.net/) * ...
What is KVM?
Is an open source virtualization technology used to operate on x86 hardware. From the official [docs](https://www.linux-kvm.org/page/Main_Page) Recommended read: * [Red Hat Article - What is KVM?](https://www.redhat.com/en/topics/virtualization/what-is-KVM)
What is Libvirt?
It's an open source collection of software used to manage virtual machines. It can be used with: KVM, Xen, LXC and others. It's also called Libvirt Virtualization API. From the official [docs](https://libvirt.org/) Hypervisor supported [docs](https://libvirt.org/drivers.html)
What the awk command does? Have you used it? What for?
From Wikipedia: "AWK is domain-specific language designed for text processing and typically used as a data extraction and reporting tool"
How to print the 4th column in a file?
`awk '{print $4}' file`
How to print every line that is longer than 79 characters?
`awk 'length($0) > 79' file`
What the lsof command does? Have you used it? What for?
What is the difference between find and locate?
How a user process performs a privileged operation, such as reading from the disk?
Using system calls
What is a system call? What system calls are you familiar with?
How a program executes a system call?
- A program executes a trap instruction. The instruction jump into the kernel while raising the privileged level to kernel space. - Once in kernel space, it can perform any privileged operation - Once it's finished, it calls a "return-from-trap" instruction which returns to user space while reducing back the privilege level to user space.
Explain the fork() system call
fork() is used for creating a new process. It does so by cloning the calling process but the child process has its own PID and any memory locks, I/O operations and semaphores are not inherited.
What is the return value of fork()?
- On success, the PID of the child process in parent and 0 in child process - On error, -1 in the parent
Name one reason for fork() to fail
Not enough memory to create a new process
Why do we need the wait() system call?
wait() is used by a parent process to wait for the child process to finish execution. If wait is not used by a parent process then a child process might become a zombie process.
How the kernel notifies the parent process about child process termination?
The kernel notifies the parent by sending the SIGCHLD to the parent.
How the waitpid() is different from wait()?
The waitpid() is a non-blocking version of the wait() function.
It also supports using library routine (e.g. system()) to wait a child process without messing up with other children processes for which the process has not waited.
True or False? The wait() system call won't return until the child process has run and exited
True in most cases though there are cases where wait() returns before the child exits.
Explain the exec() system call
It transforms the current running program into another program.
Given the name of an executable and some arguments, it loads the code and static data from the specified executable and overwrites its current code segment and current static code data. After initializing its memory space (like stack and heap) the OS runs the program passing any arguments as the argv of that process.
True or False? A successful call to exec() never returns
True
Since a successful exec replace the current process, it can't return anything to the process that made the call.
What system call is used for listing files?
What system calls are used for creating a new process?
fork(), exec() and the wait() system call is also included in this workflow.
What execve() does?
Executes a program. The program is passed as a filename (or path) and must be a binary executable or a script.
What is the return value of malloc?
Explain the pipe() system call. What does it used for?
[Unix pipe implementation](https://toroid.org/unix-pipe-implementation) "Pipes provide a unidirectional interprocess communication channel. A pipe has a read end and a write end. Data written to the write end of a pipe can be read from the read end of the pipe. A pipe is created using pipe(2), which returns two file descriptors, one referring to the read end of the pipe, the other referring to the write end."
What happens when you execute ls -l?
* Shell reads the input using getline() which reads the input file stream and stores into a buffer as a string * The buffer is broken down into tokens and stored in an array this way: {"ls", "-l", "NULL"} * Shell checks if an expansion is required (in case of ls *.c) * Once the program in memory, its execution starts. First by calling readdir() Notes: * getline() originates in GNU C library and used to read lines from input stream and stores those lines in the buffer
What happens when you execute ls -l *.log?
What readdir() system call does?
What exactly the command alias x=y does?
Why running a new program is done using the fork() and exec() system calls? why a different API wasn't developed where there is one call to run a new program?
This way provides a lot of flexibility. It allows the shell for example, to run code after the call to fork() but before the call to exec(). Such code can be used to alter the environment of the program it about to run.
Describe shortly what happens when you execute a command in the shell
The shell figures out, using the PATH variable, where the executable of the command resides in the filesystem. It then calls fork() to create a new child process for running the command. Once the fork was executed successfully, it calls a variant of exec() to execute the command and finally, waits the command to finish using wait(). When the child completes, the shell returns from wait() and prints out the prompt again.
How to create a file of a certain size?
There are a couple of ways to do that: * dd if=/dev/urandom of=new_file.txt bs=2MB count=1 * truncate -s 2M new_file.txt * fallocate -l 2097152 new_file.txt
What does the following block do?: ``` open("/my/file") = 5 read(5, "file content") ```
These system calls are reading the file
/my/file and 5 is the file descriptor number.
Describe three different ways to remove a file (or its content)
What is the difference between a process and a thread?
What is context switch?
From [wikipedia](https://en.wikipedia.org/wiki/Context_switch): a context switch is the process of storing the state of a process or thread, so that it can be restored and resume execution at a later point
You found there is a server with high CPU load but you didn't find a process with high CPU. How is that possible?
When you run ip a you see there is a device called 'lo'. What is it and why do we need it?
What the traceroute command does? How does it works?
Another common way to task this questions is "what part of the tcp header does traceroute modify?"
What is network bonding? What types are you familiar with?
How to link two separate network namespaces so you can ping an interface on one namespace from the second one?
What are cgroups?
Explain Process Descriptor and Task Structure
What are the differences between threads and processes?
Explain Kernel Threads
What happens when socket system call is used?
This is a good article about the topic: https://ops.tips/blog/how-linux-creates-sockets
You executed a script and while still running, it got accidentally removed. Is it possible to restore the script while it's still running?
What is the difference between MemFree and MemAvailable in /proc/meminfo?
MemFree - The amount of unused physical RAM in your system MemAvailable - The amount of available memory for new workloads (without pushing system to use swap) based on MemFree, Active(file), Inactive(file), and SReclaimable.
What is the difference between paging and swapping?
Explain what is OOM killer
What is a Linux distribution?
What Linux distributions are you familiar with?
What are the components of a Linux distribution?
* Kernel * Utilities * Services * Software/Packages Management
Using sed, extract the date from the following line: 201.7.19.90 - - [05/Jun/1985:13:42:99 +0000] "GET /site HTTP/1.1" 200 32421
`echo $line | sed 's/.*\[//g;s/].*//g;s/:.*//g'`
What is a Linux distribution?
* A collection of packages - kernel, GNU, third party apps, ... * Sometimes distributions store some information on the distribution in `/etc/*-release` file * For example for Red Hat distribution it will be `/etc/redhat-release` and for Amazon it will be `/etc/os-release` * `lsb_release` is a common command you can use in multiple different distributions
Name 5 commands which are two letters long
ls, wc, dd, df, du, ps, ip, cp, cd ...
What ways are there for creating a new empty file?
* touch new_file * echo "" > new_file
How `cd -` works? How does it knows the previous location?
$OLDPWD
List three ways to print all the files in the current directory
* ls * find . * echo *
How to count the number of lines in a file? What about words?
You define x=2 in /etc/bashrc and x=6 ~/.bashrc you then login to the system. What would be the value of x?
What is the difference between man and info?
A good answer can be found [here](https://askubuntu.com/questions/9325/what-is-the-difference-between-man-and-info-documentation)
Explain "environment variables". How do you list all environment variables?
What is a TTY device?
How to create your own environment variables?
`X=2` for example. But this will persist to new shells. To have it in new shells as well, use `export X=2`
What a double dash (--) mean?
It's used in commands to mark the end of commands options. One common example is when used with git to discard local changes: `git checkout -- some_file`
Wildcards are implemented on user or kernel space?
If I plug a new device into a Linux machine, where on the system, a new device entry/file will be created?
/dev
Why there are different sections in man? What is the difference between the sections?
What is User-mode Linux?
In Linux, user mode is a restricted operating mode in which a user's application or process runs. User mode is a non-privileged mode that prevents user-level processes from accessing sensitive system resources directly. In user mode, an application can only access hardware resources indirectly, by calling system services or functions provided by the operating system. This ensures that the system's security and stability are maintained by preventing user processes from interfering with or damaging system resources. Additionally, user mode also provides memory protection to prevent applications from accessing unauthorized memory locations. This is done by assigning each process its own virtual memory space, which is isolated from other processes. In contrast to user mode, kernel mode is a privileged operating mode in which the operating system's kernel has full access to system resources, and can perform low-level operations, such as accessing hardware devices and managing system resources directly.
Under which license Linux is distributed?
GPL v2