| Selector | Example | Description |
|---|---|---|
* |
* { } |
Selects all elements |
elementname |
p { } |
Selects all <p> elements |
.classname |
.intro { } |
Selects all elements with class="intro" |
#idname |
#firstname { } |
Selects the element with id="firstname" |
This document outlines the design specification for a high-performance Rust-based web crawler integrated with Apache Kafka and PostgreSQL. The crawler will act as a worker within a distributed system, consuming URLs from Kafka topics, crawling the associated web pages, and storing the results in a PostgreSQL database. Docker Compose will be utilized to manage the infrastructure, ensuring seamless deployment and orchestration of the services.
- Develop a high-performance web crawler using Rust that integrates with Kafka and PostgreSQL.
- Ensure scalable crawling of up to 10 million websites.
- π SSH keys consist of a public key and a private key
- π Keys are typically stored in
~/.ssh/ - π Private keys should be kept secure and never shared
- π€ Public keys are added to remote servers or services
| Command | Description | Example |
|---|---|---|
grep pattern file |
π Search for a pattern in a file | grep error logfile.txt |
grep -i pattern file |
π Case-insensitive search | grep -i ERROR logfile.txt |
grep -w word file |
π€ Match whole words only | grep -w error logfile.txt |
grep -v pattern file |
β Invert match (show lines that don't match) | grep -v success logfile.txt |
| Command | Description | Example |
|---|---|---|
tail file.txt |
π Display the last 10 lines of a file | tail /var/log/syslog |
tail -n 20 file.txt |
π’ Display the last 20 lines of a file | tail -n 20 /var/log/auth.log |
tail -c 100 file.txt |
π Display the last 100 bytes of a file | tail -c 100 /etc/passwd |
DISCLAIMER: This information is for educational purposes only. Always obtain proper authorization before performing any penetration testing activities. Unauthorized testing is illegal and unethical.
| Command | Description | Example |
|---|---|---|
netstat |
π Network connections, routing tables, interface statistics | netstat -tuln |
ss |
π Socket statistics | ss -tuln |
| Command | Description | Example |
|---|---|---|
cat |
π Display entire log file | cat /var/log/syslog |
less |
π View log file with pagination | less /var/log/auth.log |
tail |
π View end of log file | tail /var/log/apache2/access.log |
head |
π View beginning of log file | head /var/log/mysql/error.log |
| Command | Description | Example |
|---|---|---|
parallel echo ::: A B C |
π€ Process items in parallel | Output: A, B, C (in any order) |
parallel echo {} ::: *.txt |
π Use {} as placeholder | Echoes names of all .txt files |
cat file.txt | parallel echo |
π₯ Read input from stdin | Processes each line of file.txt |
parallel -j4 command ::: item1 item2 item3 |
π’ Limit to 4 jobs at a time | Runs 'command' on items, max 4 parallel |
| Operator | Description | Example |
|---|---|---|
; |
π Run commands sequentially | echo "Hello"; echo "World" |
&& |
β Run next command only if previous succeeds | mkdir dir && cd dir |
|| |
β Run next command only if previous fails | ping -c1 google.com || echo "Offline" |
| |
πΏ Pipe output of one command to another | ls -l | grep ".txt" |
| Command | Description |
|---|---|
ssh user@hostname |
π Connect to a remote host |
ssh -p 2222 user@hostname |
π’ Connect using a specific port |
ssh -i ~/.ssh/id_rsa user@hostname |
π Connect using a specific identity file |
exit or logout |
π Terminate the SSH session |