I had written a quick script off the top of my head, but my astute reader reminded me of the fully-featured LogParser tool that can not only extract IP addresses but can also pull out a whole host of information from your log files:
Parse IP addresses out of a web server log file
This entry was posted in Uncategorized. Bookmark the permalink. Follow any comments here with the RSS feed for this post.
9 Responses to Parse IP addresses out of a web server log file
Leave a Reply
You must be logged in to post a comment.
I’m just left wondering what your intention with this post was…
You knew there were better tools and method out there to do the job, and you knew people would call them out. So that rules out the “providing a solution for others to follow” motive.
But then you seem a bit defensive when people suggest alternatives, so that rules out the “information gathering” motive.
I can’t believe you haven’t heard of “x”. There are what? 600 different things that could have worked? You did one that you know well and wrote quickly, that’s what works well. There a couple things I don’t agree with.
1. loading all of the files into a single combined text string. I would have opened one file, processed the matches then moved on to the next file.
2. Writing and running this as a unit test. The test framework brings nothing to your solution. You aren’t using a setup or any assertions. You have a console application that you use NUnit to run. I’d have just written it as a console application.
I probably would have also output the results to a file instead of just the console window, but that’s a very minor change.
Jeffrey — Ayende’s right here. The performance implications of the following lines are really terrible:
string text = “”;
…
foreach (var filePath in filePaths) {
text += File.ReadAllText(filePath);
}
You’re allocating:
1.) A string to hold the entire contents of 1 text file
2.) Another string that holds the entire contents read up to this point PLUS the data from step #1
3.) And you’re repeating this for every log file
Like he said, you could solve this by processing a file at a time, line-by-line, but if you don’t want to do that, you could improve things tremendously by using a StringBuild that you append to.
1 word: PowerShell!
Log parser is the way to go… especially when you are talking about large log files.
http://www.microsoft.com/downloads/details.aspx?FamilyID=890cd06b-abf8-4c25-91b2-f8d975cf8c07&displaylang=en
Walk, don’t run, to http://www.logparser.com, and query your logs like you would if they were SQL tables …
You asked for it.
require ‘set’
require ‘pp’
results = SortedSet.new
Dir['*.log'].each do |path|
File.read(path).scan(
/\b(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\b/
).flatten.each { |ip| results < < ip }
end
pp results
It took me 2 minutes to write this (I’m slow
). Not that you took any longer. I’ve just been thinking lately how it feels like I’m at the point where I can constantly throw away little solutions like this and not get wrapped up in reusability for a utility because they’re one of the few things I can (almost) write faster in my head than I can type.
That level of proficiency or approach to a problem doesn’t get much press in programming circles it seems. Then again, not much more to say, so that’s probably just because it’s not a very interesting topic. Just thinking out loud here.
@Ayende,
Just enough to find out what IP addresses have accessed the server in the past few days.
Perhaps, but it works great for 10-20 files.
This is horribly inefficient for large files or large amount of files.
It would be better to do this on a line by line basis.