JIYIK CN >

Linux classic practice - log filtering

Let's talk about the problem first. Count the number of IP addresses in a log file after deduplication. In fact, this is a very common and relatively simple problem. I personally think that the most important thing should be matching IP addresses. The rest is a matter of proficiency in Linux commands.
First of all, here I will talk about the command I used to solve this problem.

• grepUsed to retrieve the IP address in the log file;
• uniqUsed to deduplicate the retrieved IP addresses;
• wcUsed to count the number of IP addresses;

Next we mainly introduce grephow to match the IP address

grepAs a frequently used command in Linux, it is a member of the pipeline command like the cut command. And its function is also to analyze a line of data and extract the data we want from the analyzed data. It is equivalent to a search function. Of course, grep is much more powerful than cut. Grep has a variety of search conditions, and can even cooperate with regular expressions to search. Here we mainly use its regular expressions.

Among them, we mainly use the following options of grep

• -iCase-insensitive matching;
• -oShow only matching strings
• -ESpecifying a regular expression

For IP addresses, the value range is 0.0.0.0-255.255.255.255. So we can first match 0-255, and then repeat the regular expression four times.

Let's first look at the regular expression that matches 0-255.

(2[0-5]{2}|(1)?(?(\2)[0-9]{2}|([1-9][0-9]|[0-9])))

Here we break down this regular expression. First, for the case where the first digit may be 2: 200-255. So we can use the regular expression: 2[0-5]{2}. Then for the matching 100-199case, the first digit must be judged whether it is 1. But here we have to discuss it separately. If it is three digits, it is 100-199; if it is two digits, it is a match 10-99, and if it is one digit, it is a match 0-9. So here the first digit 1 can be optional, and then the existence of 1 is used as a condition to judge whether it matches 100-199. So we also need a条件子组

Let's look at the regular expression

((1)?(?(\2)[0-9]{2}|([1-9][0-9]|[0-9])))

If 1 is captured, the condition (?(\2)is true, then it is followed by yes [0-9]{2}; otherwise (?(\2), it is false else-pattern, and an optional path ([1-9][0-9]|[0-9])can be used to match 10-99or 0-9.

So in summary,

(2[0-5]{2}|(1)?(?(\2)[0-9]{2}|([1-9][0-9]|[0-9])))

Then add a dot in front and repeat 3 times

(\.(2[0-5]{2}|(1)?(?(\2)[0-9]{2}|([1-9][0-9]|[0-9])))){3}

Finally, the entire regular expression is

(2[0-5]{2}|(1)?(?(\2)[0-9]{2}|([1-9][0-9]|[0-9])))(\.(2[0-5]{2}|(1)?(?(\2)[0-9]{2}|([1-9][0-9]|[0-9])))){3}

By grepadding the above regular expression to match all IP addresses; then use uniqthe command to remove duplicates; and finally use it wcto perform statistics. The entire command is as follows

$ grep -ioE '(2[0-5]{2}|(1)?(?(\2)[0-9]{2}|([1-9][0-9]|[0-9])))(\.(2[0-5]{2}|(1)?(?(\2)[0-9]{2}|([1-9][0-9]|[0-9])))){3}' | uniq -c | wc -l

The above problem is just an application of grep using regular expressions. There can be many variations of this problem, such as counting the number of IP address visits and then sorting them, finding the top ten IP addresses with the most visits, etc.