Perl script to extract info from a log file

[jawn]

I wrote a script to extract information from the bazaar log file. The output is a list of all items sold, buyers, totals for the items, and a grand total at the end of the list.

Here is an example of the output with the buyer's names sanitized. The grand total does not add up because I extracted only two items from the output list.

Tear Etched Swatch

	 Qty     Amount    Buyer
	-----  ----------  ------------------------
	   43     430.000  Name #1
	    6      60.000  Name #2
	    4      40.000  Name #3
	   27     270.000  Name #4
	   52     520.000  Name #5
	   73     730.000  Name #6
	   17     170.000  Name #7
	-----  ----------
	  222    2220.000

Wok

	 Qty     Amount    Buyer
	-----  ----------  ------------------------
	    1      35.000  Name #1
	    1      35.000  Name #2
	    1      35.000  Name #3
	-----  ----------
	    3     105.000

Grand Total

	 Qty     Amount    
	-----  ----------
	11543  330570.200

I have not seen much reaction to the other perl script I have posted, and so far only one person has sent a PM to me asking for my bought and sold scripts mentioned in the jewelcraft forum about the cost to raise JC from 0 to 250.

So I am not sure if there is much interest in the scripts, and I don't want to waste the time of the EQTraders community posting the scripts if there is no interest. Please let me know.

For those who don't know, there is a Perl for Microsoft Windows. You can download it from http://www.activestate.com/Products/ActivePerl

I have no affiliation with Active State. I downloaded ActivePerl and installed it on my Windows XP system to test it. It works. Normally, I use Perl on my Linux system.

[andyhre]

I am VERY interested, as I am developing an academic paper on pricing behavior in the bazaar.

I am not sufficiently computer savvy to use this, but I have friends who will help me.

You may have seen me ask this, but can you think of any way to capture and parse the information one gets when one does a search in /bazaar mode?

[Lothay]

I for one liked your script.

I'm a PERL programmer myself, so you saved me the time of writing one, and you thought of something I wouldn't have... namely the chat logging for commentary.

Thank you.

[Lilosh]

If there would be a way to form this into an Executable file that analyzed a Bazaar log in the same directroy, I'd be all over that.


If I had to compile code, and form my own executable, I'm not sure how competant I'd be.


-Lilosh

[Inyidd]

Perl isn't a compiled language generally. You would download the Perl interpreter (the Active Perl link) and then run the perl script through the interpreter (i.e., "perl myperlscript.pl").

[Nihixul]

great job! i am quite interested in such parsers, but have so far not bothered writing one. thanks for the contribution!

[Lothay]

ActiveState has a tool for converting a PERL script to an executable. Use at your own risk, I'm not qulaified to explain all the complexities.

[Inyidd]

That's kinda why I said generally.

[jawn]

Program: eqbzrsold.pl

Description: List items sold as trader with quantities, buyers, and totals

Usage: eqbzrsold.pl LOG-FILE > REPORT-FILE

Edit:
Edited to fix the bug pointed out by sansaan.

#!/usr/local/bin/perl

%coin2ix = (qw/p 0 g 2 s 3 c 4/);
%soldamt = ();
%soldqty = ();

$soldre = '\[
	[A-Z][a-z][a-z]		# Day abbreviation
	\s
	[A-Z][a-z][a-z]		# Month abbreviation
	\s
	\d\d			# Day number
	\s
	\d\d:\d\d:\d\d		# Time
	\s
	\d\d\d\d		# Year
	\]
	\s
	(.*)			# Buyer
	\spurchased\s
	(\d+)			# Quantity
	\s
	(.*)			# Item
	\sfor\s\(\s
	(.*)			# Price
	\)\.';

while (<>) {
    chomp;
    if ($_ =~ /$soldre/x) {
	$buyer = $1;
	$qty = $2;
	$itemname = $3;
	$price = $4;
	$amt = long_price_2_short_price($price);
	$soldkey = join("\t", $itemname, $buyer);
	$soldqty{$soldkey} += $qty;
	$soldamt{$soldkey} += $amt;
    }
}

$totalamt  = 0; 
$totalqty  = 0;
$itemamt   = 0; 
$itemqty   = 0;
$fencepost = 0;
$curritem  = "";
foreach $soldkey (sort {$a cmp $b} keys %soldqty) {
    ($itemname, $buyer) = split("\t", $soldkey);
    if ($itemname ne $curritem) {
	if ($fencepost) {
	    print STDOUT "\t-----  ----------\n";
	    printf STDOUT ("\t%5d  %10.3f\n", $itemqty, $itemamt);
	    $totalqty += $itemqty;
	    $totalamt += $itemamt;
	    $itemqty = 0;
	    $itemamt = 0;
	    print STDOUT "\n";
	}
	print STDOUT ($itemname, "\n\n");
	print STDOUT "\t Qty     Amount    Buyer\n";
	print STDOUT "\t-----  ----------  ------------------------\n";
	$curritem = $itemname;
    }
    $fencepost = 1;
    printf STDOUT ("\t%5d  %10.3f  %s\n",
		   $soldqty{$soldkey}, $soldamt{$soldkey}, $buyer);
    $itemqty += $soldqty{$soldkey};
    $itemamt += $soldamt{$soldkey};
}

# Print total for last item.
print STDOUT "\t-----  ----------\n";
printf STDOUT ("\t%5d  %10.3f\n", $itemqty, $itemamt);
$totalqty += $itemqty;
$totalamt += $itemamt;

# Print total for all items.
print STDOUT "\nGrand Total\n\n";
print STDOUT "\t Qty     Amount    \n";
print STDOUT "\t-----  ----------\n";
printf STDOUT ("\t%5d  %10.3f\n", $totalqty, $totalamt);


sub long_price_2_short_price {
    my $name;
    my $value;
    my $price = shift;
    my @fields = split(' ', $price);
    my $count = scalar @fields;
    if ($count < 1 or
	$count > 4) {
	print STDERR "Something is wrong with the price field: $price\n";
	exit(-1);
    }
    @cointab = (0, ".", 0, 0, 0);
    foreach (@fields) {
	($value, $name) = /(\d+)([pgsc])/;
	$cointab[$coin2ix{$name}] = $value;
    }
    return join('', @cointab);
}

[Gnomtrix]

Interested Much

I'm always interested in anything which can parse any part of EQ, mostly for kicks, and sometimes for useful stuff.

I don't sell enough stuff on my trader to make this script essential, but it will be nice to track what gets sold without having to go through the log file or sell window manually (esp. when I'm selling stuff for friends).

[jawn]

This next script extracts price information from a log file. The script prints a list of items with their price, sorted by zone, merchant, and item. It is possible to get more than one price for the same item from the same merchant. For example, if you browse the merchant and you forgot to equip your opal encrusted steins. This script will report buy and sell prices.

Also, traders in bazaar are almost exactly like NPC merchants. You cannot sell to traders in bazaar, but this script will report trader selling prices.

Click on each item for which you want the price captured so that it is entered into your log file. You do not have to buy nor sell the item.

Here is some sample output from the script (with some of the nicer things I have found):

Cost to buy items from Channeler Keatrem in Plane of Knowledge
      9.765  Bag of Sewn Evil-Eye
      1.050  Coiled Spring

Cost to buy items from Channeler Olaemos in Plane of Knowledge
      1.049  Coiled Spring
      1.050  Deluxe Sewing Kit
     10.709  Spell: Draught of Fire
     12.991  Spell: Draught of Fire

The script will be in the next reply.

For those people who are interested in the performance of a perl script, I tested this on all the log files that I have for all my characters. The total size of all my logs thus far is 444,991,921 bytes. It took almost three minutes to process all the files.

eqlogprices.pl /tmp/eql/* > eql.prices

Here are the results from two tests:

Mon Nov 10 14:32:46 EST 2003
163.37user 3.56system 2:48.19elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (109159major+1673minor)pagefaults 0swaps
Mon Nov 10 14:35:35 EST 2003

Mon Nov 10 14:48:57 EST 2003
163.54user 3.35system 2:47.96elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (109159major+1673minor)pagefaults 0swaps
Mon Nov 10 14:51:45 EST 2003

[jawn]

Program: eqlogprices.pl

Description: List buy and sell prices for items on merchants and traders

Usage: eqlogprices.pl LOG-FILE > REPORT-FILE

#!/usr/local/bin/perl

$debug = 1;
$currzone = "an unknown zone";
%coin2ix = (qw/platinum 0 gold 2 silver 3 copper 4/);
%cost_price = ();
%sell_price = ();

$zonere = '\[
	[A-Z][a-z][a-z]		# Day abbreviation
	\s
	[A-Z][a-z][a-z]		# Month abbreviation
	\s
	\d\d			# Day number
	\s
	\d\d:\d\d:\d\d		# Time
	\s
	\d\d\d\d		# Year
	\]
	\sYou\shave\sentered\s
	(.*)
	\.';


$costre = '\[
	[A-Z][a-z][a-z]		# Day abbreviation
	\s
	[A-Z][a-z][a-z]		# Month abbreviation
	\s
	\d\d			# Day number
	\s
	\d\d:\d\d:\d\d		# Time
	\s
	\d\d\d\d		# Year
	\]
	\s
	(.*)
	\stells\syou,\s\'That\'ll\sbe\s
	(.*)
	\s(?:per|for\sthe)\s
	(.*)
	(?:,\sand\sthe\srent\swill\sbe\s.*\sper\s.*|)
	\.\'';

$sellre = '\[
	[A-Z][a-z][a-z]		# Day abbreviation
	\s
	[A-Z][a-z][a-z]		# Month abbreviation
	\s
	\d\d			# Day number
	\s
	\d\d:\d\d:\d\d		# Time
	\s
	\d\d\d\d		# Year
	\]
	\s
	(.*)
	\stells\syou,\s\'I\'ll\sgive\syou\s
	(.*)
	\s(?:per|for\sthe)\s
	(.*)
	\.\'';

# Somewhere between Aug 23, 2003 and Aug 27, 2003, what Inkeeper Freegraze
# says to you regarding the cost of Room Key # 1 changed.  Here is an
# example of before and after.
#
# [Sat Aug 23 17:33:10 2003] Inkeeper Freegraze tells you,
# 'That'll be 4 silver 1 copper for the Room Key # 1, and the rent will be 3 copper per month.'
#
# [Wed Aug 27 03:59:01 2003] Inkeeper Freegraze tells you,
# 'That'll be 4 silver 1 copper for the Room Key # 1.'
#
# This caused a problem for my script originally, so I put in the following
# substitute in the while loop, to eliminate the rent portion of the line.
#
# s/, and the rent will be .* per .*\.'/.'/;

while (<>) {
    chomp;
    if ($_ =~ /$zonere/x) {
	$currzone = $1;
	next;
    }
    s/, and the rent will be .* per .*\.'/.'/;
    if ($_ =~ /$costre/x) {
	$merchant = $1;
	$price = $2;
	$itemname = $3;
	$currmerchant = $merchant;
	$currcost = long_price_2_short_price($price);
	$curritem = $itemname;
	$costkey = join("\t", $currzone, $merchant, $itemname, $currcost);
	$cost_price{$costkey} = 1;
	next;
    }
    if ($_ =~ /$sellre/x) {
	$merchant = $1;
	$price = $2;
	$itemname = $3;
	$currmerchant = $merchant;
	$currsell = long_price_2_short_price($price);
	$curritem = $itemname;
	$sellkey = join("\t", $currzone, $merchant, $itemname, $currsell);
	$sell_price{$sellkey} = 1;
	next;
    }
}

$blankflag = 0;
$currzone = "";
$currmerchant = "";
foreach $costkey (sort {$a cmp $b} keys %cost_price) {
    ($zone, $merchant, $itemname, $cost) = split("\t", $costkey);
    if ($zone ne $currzone or $merchant ne $currmerchant) {
	print STDOUT "\n" if $blankflag;
	$blankflag = 1;
	print STDOUT ("Cost to buy items from ", $merchant, " in ", $zone, "\n");
	$currzone = $zone;
	$currmerchant = $merchant;
    }
    printf STDOUT ("%11s  %s\n", $cost, $itemname);
}

$currzone = "";
$currmerchant = "";
foreach $sellkey (sort {$a cmp $b} keys %sell_price) {
    ($zone, $merchant, $itemname, $sell) = split("\t", $sellkey);
    if ($zone ne $currzone or $merchant ne $currmerchant) {
	print STDOUT "\n" if $blankflag;
	$blankflag = 1;
	print STDOUT ("Price to sell items to ", $merchant, " in ", $zone, "\n");
	$currzone = $zone;
	$currmerchant = $merchant;
    }
    printf STDOUT ("%11s  %s\n", $sell, $itemname);
}

# [Sat May 24 03:01:45 2003]
# Scon McDaniel tells you, 'I'll give you absolutely nothing for the Third Riddle for the Troll.'
#
# Changed the test for "absolutely nothing" from "eq" to "=~"
# because of the above log entry.

sub long_price_2_short_price {
    my $name;
    my $value;
    my $price = shift;
    return 0.000 if ($price eq "0 money");
    return 0.000 if ($price =~ /absolutely nothing/);
    my @fields = split(' ', $price);
    my $count = scalar @fields;
    if ($count != 2 and
	$count != 4 and
	$count != 6 and
	$count != 8) {
	print STDERR "Something is wrong with the price field: $price\n";
	exit(-1);
    }
    @cointab = (0, ".", 0, 0, 0);
    while (scalar @fields) {
	$value = shift @fields;
	$name = shift @fields;
	$cointab[$coin2ix{$name}] = $value;
    }
    return join('', @cointab);
}

[YurienStonebow]

ok i'll admit that when it comes to computers i know enough to make them melt down and to paly games. otherwise i'm lost. what exactly is the purpose/usefulness of a script like this? i know you wrote it for a reason so please tell me what it is in laymans terms plz :>

[Lothay]

The point to the script is a nicely formatted, easy-to-read report of what business your trader is doing.

[Anaxo]

A script which is very useful if you sell mass quantities of items, like Celestial Essence. I've found that people who buy in bulk by the couple hundreds over a period of time are usually working on the Aid Grimel quest, and contacting them is a good way to get repeat business.